Instruction: Pytorch custom randomcrop for semantic segmentation

Input:
I am trying to implement a custom dataset loader. Firstly I resize the images and labels with the same ratio between (0.98, 1,1) then I randomly crop both images and labels with same parameters so that I can feed them into NN. However, I am getting an error from PyTorch functional. Here is my code:
class RandomCrop(object):

    def __init__(self, size, padding=None, pad_if_needed=True, fill=0, padding_mode='constant'):

        self.size = size
        self.padding = padding
        self.pad_if_needed = pad_if_needed
        self.fill = fill
        self.padding_mode = padding_mode

    @staticmethod
    def get_params(img, output_size):

        w, h = img.size
        th, tw = output_size
        if w == tw and h == th:
            return 0, 0, h, w

        i = random.randint(0, h - th)
        j = random.randint(0, w - tw)
        return i, j, th, tw

    def __call__(self, data):

        img,mask = data["image"],data["mask"]

        # pad the width if needed
        if self.pad_if_needed and img.size[0] < self.size[1]:
            img = F.pad(img, (self.size[1] - img.size[0], 0), self.fill, self.padding_mode)
            mask = F.pad(mask, (self.size[1] - mask.size[0], 0), self.fill, self.padding_mode)
        # pad the height if needed
        if self.pad_if_needed and img.size[1] < self.size[0]:
            img = F.pad(img, (0, self.size[0] - img.size[1]), self.fill, self.padding_mode)
            mask = F.pad(mask, (0, self.size[0] - mask.size[1]), self.fill, self.padding_mode)

        i, j, h, w = self.get_params(img, self.size)
        crop_image = transforms.functional.crop(img, i, j, h, w)
        crop_mask = transforms.functional.crop(mask, i, j, h, w)

        return{"image": crop_image, "mask": crop_mask }

Here is the error:
AttributeError: 'Image' object has no attribute 'dim'

Output: Mistakenly I imported nn.functional.pad instead of the transforms.functional.pad. After changing it everything went smoothly


Instruction: Defining python class method using arguments from __init__

Input:
What I want to achieve
I want to define a class with two modes A and B, so that the forward method of the class changes accordingly.
class MyClass():
    def __init__(self, constant):
        self.constant=constant

    def forward(self, x1,x2,function):
        if function=='A':
            return x1+self.constant

        elif function=='B':
            return x1*x2+self.constant

        else:
            print('please provide the correct function')
model1 = MyClass(2)
model1.forward(2, None, 'A')
output>>>4
model2 = MyClass(2)
model2.forward(2, 2, 'B')
output>>>6

It works, but it is not optimal, since every time when calling the forward method, it will check which function to use. However, the forward function is already set and will never be changed once the class is define, therefore, checking which function to use inside forward is super redundant in my case. (For those who notice this, I am writing my neural network model using PyTorch, two models share 90% of the network architecture, the only 10% differences is the way they do feedforward).
My desired version
I want to set the forward method when the class is defined, so that I can achieve this
model1 = MyClass(2, 'A')
model1.forward(2)
output>>>4

model2 = MyClass(2, 'B')
model2.forward(2, 2)
output>>>6

So I rewrote my class to be:
class MyClass():
    def __init__(self, constant, function):
        self.constant=constant # There would be a lot of shared parameters for the two methods
        self.function=function # This controls the feedforward method of this class

    if self.function=='A':
        def forward(self, x1):
            return x1+self.constant

    elif self.function=='B':
        def forward(self, x1, x2):
            return x1*x2+self.constant

    else:
        print('please provide the correct function')

However, it gives me the following error.

NameError: name 'self' is not defined

How do I write the class for that it defines different forward method based on the args from __init__?

Output:
You have been trying to redefine the class with your code, such that each new object would change the definition of forward for all objects, before and after.
Fortunately, you didn't figure out how to do that.
Instead, make the chosen function an attribute of the object.  Code the two functions you want, and then assign the desired variant as you create each instance.
class MyClass():
    def __init__(self, constant, function):
        self.constant=constant
        if function == 'A':
            self.forward = self.forwardA
        elif function=='B':
            self.forward = self.forwardB
        else:
            print('please provide the correct function')

    def forwardA(self, x1):
        return x1+self.constant

    def forwardB(self, x1, x2):
        return x1*x2+self.constant

# Main
model1 = MyClass(2, 'A')
print(model1.forward(2))

model2 = MyClass(2, 'B')
print(model2.forward(2, 2))

Output:
4
6


Instruction: Unable to load model from checkpoint in Pytorch-Lightning

Input:
I am working with a U-Net in Pytorch Lightning. I am able to train the model successfully but after training when I try to load the model from checkpoint I get this error:
Complete Traceback:
Traceback (most recent call last):
  File "src/train.py", line 269, in <module>
    main(sys.argv[1:])
  File "src/train.py", line 263, in main
    model = Unet.load_from_checkpoint(checkpoint_callback.best_model_path)
  File "/home/africa_wikilimo/miniconda3/envs/xarray_test/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 153, in load_from_checkpoint
    model = cls._load_model_state(checkpoint, *args, strict=strict, **kwargs)
  File "/home/africa_wikilimo/miniconda3/envs/xarray_test/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 190, in _load_model_state
    model = cls(*cls_args, **cls_kwargs)
  File "src/train.py", line 162, in __init__
    self.inc = double_conv(self.n_channels, 64)
  File "src/train.py", line 122, in double_conv
    nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
  File "/home/africa_wikilimo/miniconda3/envs/xarray_test/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 406, in __init__
    super(Conv2d, self).__init__(
  File "/home/africa_wikilimo/miniconda3/envs/xarray_test/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 50, in __init__
    if in_channels % groups != 0:
TypeError: unsupported operand type(s) for %: 'dict' and 'int'

I tried surfing the github issues and forums, am not able to figure out what the issue is. Please help.
Here's the code of my model and the checkpoint loading step:
Model:
class Unet(pl.LightningModule):
    def __init__(self, n_channels, n_classes=5):
        super(Unet, self).__init__()
        # self.hparams = hparams

        self.n_channels = n_channels
        self.n_classes = n_classes
        self.bilinear = True
        self.logger = WandbLogger(name="Adam", project="pytorchlightning")

        def double_conv(in_channels, out_channels):
            return nn.Sequential(
                nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
                nn.BatchNorm2d(out_channels),
                nn.ReLU(inplace=True),
                nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1),
                nn.BatchNorm2d(out_channels),
                nn.ReLU(inplace=True),
            )

        def down(in_channels, out_channels):
            return nn.Sequential(
                nn.MaxPool2d(2), double_conv(in_channels, out_channels)
            )

        class up(nn.Module):
            def __init__(self, in_channels, out_channels, bilinear=False):
                super().__init__()

                if bilinear:
                    self.up = nn.Upsample(
                        scale_factor=2, mode="bilinear", align_corners=True
                    )
                else:
                    self.up = nn.ConvTranspose2d(
                        in_channels // 2, in_channels // 2, kernel_size=2, stride=2
                    )

                self.conv = double_conv(in_channels, out_channels)

            def forward(self, x1, x2):
                x1 = self.up(x1)
                # [?, C, H, W]
                diffY = x2.size()[2] - x1.size()[2]
                diffX = x2.size()[3] - x1.size()[3]

                x1 = F.pad(
                    x1, [diffX // 2, diffX - diffX // 2, diffY // 2, diffY - diffY // 2]
                )
                x = torch.cat([x2, x1], dim=1)
                return self.conv(x)

        self.inc = double_conv(self.n_channels, 64)
        self.down1 = down(64, 128)
        self.down2 = down(128, 256)
        self.down3 = down(256, 512)
        self.down4 = down(512, 512)
        self.up1 = up(1024, 256)
        self.up2 = up(512, 128)
        self.up3 = up(256, 64)
        self.up4 = up(128, 64)
        self.out = nn.Conv2d(64, self.n_classes, kernel_size=1)

    def forward(self, x):
        x1 = self.inc(x)

        x2 = self.down1(x1)

        x3 = self.down2(x2)

        x4 = self.down3(x3)

        x5 = self.down4(x4)

        x = self.up1(x5, x4)
        x = self.up2(x, x3)
        x = self.up3(x, x2)
        x = self.up4(x, x1)

        return self.out(x)

    def training_step(self, batch, batch_nb):
        x, y = batch

        y_hat = self.forward(x)
        loss = self.MSE(y_hat, y)

        # wandb_logger.log_metrics({"loss":loss})
        return {"loss": loss}

    def training_epoch_end(self, outputs):
        avg_train_loss = torch.stack([x["loss"] for x in outputs]).mean()
        self.logger.log_metrics({"train_loss": avg_train_loss})
        return {"average_loss": avg_train_loss}

    def test_step(self, batch, batch_nb):
        x, y = batch
        y_hat = self.forward(x)
        loss = self.MSE(y_hat, y)
        return {"test_loss": loss, "pred": y_hat}

    def test_end(self, outputs):

        avg_loss = torch.stack([x["test_loss"] for x in outputs]).mean()

        return {"avg_test_loss": avg_loss}

    def MSE(self, logits, labels):

        return torch.mean((logits - labels) ** 2)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.1, weight_decay=1e-8)

Main Function:
def main(expconfig):
    # Define checkpoint callback
    checkpoint_callback = ModelCheckpoint(
        filepath="/home/africa_wikilimo/data/model_checkpoint/",
        save_top_k=1,
        verbose=True,
        monitor="loss",
        mode="min",
        prefix="",
    )

    # Initialise datasets
    print("Initializing Climate Dataset....")
    clima_train = Clima_Dataset(expconfig[0])

    # Initialise dataloaders
    print("Initializing train_loader....")
    train_dataloader = DataLoader(clima_train, batch_size=2, num_workers=4)

    # Initialise model and trainer
    print("Initializing model...")
    model = Unet(n_channels=9, n_classes=5)
    print("Initializing Trainer....")
    if torch.cuda.is_available():

        model.cuda()

        trainer = pl.Trainer(
            max_epochs=1,
            gpus=1,
            checkpoint_callback=checkpoint_callback,
            early_stop_callback=None,
        )
    else:

        trainer = pl.Trainer(max_epochs=1, checkpoint_callback=checkpoint_callback)

    trainer.fit(model, train_dataloader=train_dataloader)
    print(checkpoint_callback.best_model_path)
    model = Unet.load_from_checkpoint(checkpoint_callback.best_model_path)

Output:
Cause
This happens because your model is unable to load hyperparameters(n_channels, n_classes=5) from the checkpoint as you do not save them explicitly.
Fix
You can resolve it by using the self.save_hyperparameters('n_channels', 'n_classes')method in your Unet class's init method.
Refer PyTorch Lightning hyperparams-docs for more details on the use of this method. Use of save_hyperparameters lets the selected params to be saved in the hparams.yaml along with the checkpoint.
Thanks @Adrian Wälchli
(awaelchli) from the PyTorch Lightning core contributors team who suggested this fix, when I faced the same issue.


Instruction: Fill regions from contour based on cumsum

Input:
I got an image for scene classification:

And using cumsum, I want to segment the three parts of it.
I take this simple operation in pytorch (can use tensorflow of course too, or python)

Output:
You can use torch.flipud to perform the cumsum in both directions:
mask = (src_img.cumsum(dim=0) >0 ) + 2* torch.flipud(torch.flipud(src_img).cumsum(dim=0)>0)

Resulting with:


Instruction: What does PyTorch classifier output?

Input:
So i am new to deep learning and started learning PyTorch. I created a classifier model with following structure.
class model(nn.Module):
def __init__(self):
    super(model, self).__init__()
    resnet = models.resnet34(pretrained=True)
    layers = list(resnet.children())[:8]
    self.features1 = nn.Sequential(*layers[:6])
    self.features2 = nn.Sequential(*layers[6:])
    self.classifier = nn.Sequential(nn.BatchNorm1d(512), nn.Linear(512, 3))

def forward(self, x):
    x = self.features1(x)
    x = self.features2(x)
    x = F.relu(x)
    x = nn.AdaptiveAvgPool2d((1,1))(x)
    x = x.view(x.shape[0], -1)
    return self.classifier(x)

So basically I wanted to classify among three things {0,1,2}. While evaluating, I passed the image it returned a Tensor with three values like below
    (tensor([[-0.1526,  1.3511, -1.0384]], device='cuda:0', grad_fn=<AddmmBackward>)

So my question is what are these three numbers? Are they probability ?
P.S. Please pardon me If I asked something too silly.

Output:
The final layer nn.Linear (fully connected layer) of self.classifier of your model produces values, that we can call a scores, for example, it may be: [10.3, -3.5, -12.0], the same you can see in your example as well: [-0.1526,  1.3511, -1.0384]  which are not normalized and cannot be interpreted as probabilities.
As you can see it's just a kind of "raw unscaled" network output, in other words these values are not normalized, and it's hard to use them or interpret the results, that's why the common practice is converting them to normalized probability distribution by using softmax after the final layer, as @skinny_func has already described. After that you will get the probabilities in the range of 0 and 1, which is more intuitive representation.


Instruction: Neural Networks & ML without GPU. what are my options?

Input:
I am starting working with Neural Networks, ML etc. I don't have a GPU, what are my options?
Are Pytorch and Tensorflow good options? If so, which are the pros and cons of each, and which should I choose?
I will be using Python and Linux (ubuntu 20.04) if that makes any difference.
(I cant afford a GPU or cloud services because im broke and 15 y.o.)

Output:
It depends what you want to do. If you are working on images, you will need GPU. If you are working on other type of data, with classification or regression, you can use your CPU.
Working on Ubuntu is perfect, it is my choice too.
Now for Neural Networks, Tensorflow and Pytorch ARE the options, you may try any of them. I worked mostly on Tensorflow.
If you want to use GPU, you may try the Kaggle competitions where you may have a good amount of GPU every week. You may aslo try Google Colab notebooks, where you may have an amount of GPU, you have to connect with a Google account and eventually connect your notebook to your drive, where you can stock your data.


Instruction: Why can't I use Cross Entropy Loss for multilabel?

Input:
I'm in the process of finetuning a BERT model to the long answer task in the Natural Questions dataset. I'm training the model just like a SQuAD model (predicting start and end tokens).
I use Huggingface and PyTorch.
So the targets and labels have a shape/size of [batch, 2]. My problem is that I can't input "multi-targets" which I think is refering to the fact that the last shape is 2.

RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:18

Should I choose another loss function or is there another way to bypass this problem?
This code I'm using:
def loss_fn(preds, targets):
    return nn.CrossEntropyLoss()(preds,labels)

class DecoderModel(nn.Module):

    def __init__(self, model_args, encoder_config, loss_fn):
        super(DecoderModel, self).__init__()
        # ...

    def forward(self, pooled_output, labels):
        pooled_output = self.dropout(pooled_output)
        logits = self.linear(pooled_output)

        start_logits, end_logits = logits.split(1, dim = -1)
        start_logit = torch.squeeze(start_logits, axis=-1)
        end_logit = torch.squeeze(end_logits, axis=-1)

        # Concatenate into a "label"
        preds = torch.cat((start_logits, end_logits), -1)

        # Calculate loss
        loss = self.loss_fn(
            preds = preds,
            labels = labels)

        return loss, preds

The targets properties are:
torch.int64 & [3,2]
The predictions properties are:
torch.float32 & [3,2]
SOLVED - this is my solution
def loss_fn(preds:list, labels):
    start_token_labels, end_token_labels = labels.split(1, dim = -1)
    start_token_labels = start_token_labels.squeeze(-1)
    end_token_labels = end_token_labels.squeeze(-1)

    print('*'*50)
    print(preds[0].shape) # preds [0] and [1] has the same shape and dtype
    print(preds[0].dtype) # preds [0] and [1] has the same shape and dtype
    print(start_token_labels.shape) # labels [0] and [1] has the same shape and dtype
    print(start_token_labels.dtype) # labels [0] and [1] has the same shape and dtype

    start_loss = nn.CrossEntropyLoss()(preds[0], start_token_labels)
    end_loss = nn.CrossEntropyLoss()(preds[1], end_token_labels)

    avg_loss = (start_loss + end_loss) / 2
    return avg_loss

Basically I'm splitting the logits (just not concatinating them) and the labels. I then do Cross Entropy loss on both of them and at last taking the average loss between the two. Hope this gives you an idea to solve your own problem!

Output:
You should not give 1-hot vectors to CrossEntropyLoss, rather the labels directly

Target: (N) where each value is 0≤targets[i]≤C−1 , or (N, d_1, d_2, ..., d_K) with K≥1 in the case of K-dimensional loss.

You can reproduce  your error looking at the docs:
>>> loss = nn.CrossEntropyLoss()
>>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.empty(3, dtype=torch.long).random_(5)
>>> output = loss(input, target)
>>> output.backward()

but if you change target to target = torch.empty((3, 5), dtype=torch.long).random_(5) then you get  the error:

RuntimeError: 1D target tensor expected, multi-target not supported

Use nn.BCELoss with logits as inputs instead, see this example: https://discuss.pytorch.org/t/multi-label-classification-in-pytorch/905/41
>>> nn.BCELoss()(torch.softmax(input, axis=1), torch.softmax(target.float(), axis=1))
>>> tensor(0.6376, grad_fn=<BinaryCrossEntropyBackward>)


Instruction: PyTorch expected CPU got CUDA tensor

Input:
I've been struggling to find what's wrong in my code. I'm trying to implement DCGAN paper and from the past 1 hour, I'm going through these errors. Could anyone please help me fix this?
I'm training this on Google colab with GPU runtime but I'm getting this error. Yesterday, I implemented the first GAN paper by Ian Goodfellow and I did not got this error. I don't know what's happening any help would be appreciated. Also, please check whether the gen_input is correct or not.
Here is the code:
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torchvision
import torchvision.transforms as transforms
from torchvision.utils import save_image
import torch.optim as optim

#---------------configuration part------------------#
lr = 0.00002 #learning rate
nc = 3 #color channels
nz = 100 #size of latent vector or size of generator input
ngf = 64 #size of feature maps in generator
ndf = 64 #size of feature maps in discriminator
height = 128 #height of the image
width = 128 #width of the image
num_epochs = 100 #the variable name tells everything
workers = 2 #number of workers to load the data in batches
batch_size = 64 #batch size
image_size = 128 #resizing parameter
root = '/content/gdrive/My Drive/sharingans/' #path to the training directory
beta1 = 0.4

#---------------------------------------------------#
#define the shape of the image
img_shape = (nc, height, width)

#---------------------------------------------------#
#define the weights initialization function
#in the DCGAN paper they state that all weights should be
#randomly initialize weights from normal distribution
#the following function does that

def weights_init(m):
    classname = m.__class__.__name__ #returns the class name(eg: Conv2d or ConvTranspose2d)
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02) #0.0 is mean and 0.02 is standard deviation
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1, 0.02) #1 is mean and 0.02 is standard deviation
        nn.init.constant_(m.bias.data, 0.0)

#---------------------------------------------------#

#implement the data loader function to load images

def load_data(image_size, root):
    transform = transforms.Compose([
        transforms.Resize(image_size),
        transforms.ToTensor(),
        transforms.Normalize((0.486, 0.486, 0.486), (0.486, 0.486, 0.486))
        ])

    train_set = torchvision.datasets.ImageFolder(root = root, transform = transform)

    return train_set
train_set = load_data(128, root)
#getting the batches of data
train_data = torch.utils.data.DataLoader(train_set, batch_size = batch_size, shuffle = True, num_workers = workers)

#---------------------------------------------------#
#implement the generator network

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.convt1 = nn.ConvTranspose2d(in_channels = nz, out_channels = ngf*8, kernel_size = 4, stride = 1, padding = 0, bias = False)
        self.convt2 = nn.ConvTranspose2d(in_channels = ngf*8, out_channels = ngf*4, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt3 = nn.ConvTranspose2d(in_channels = ngf*4, out_channels = ngf*2, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt4 = nn.ConvTranspose2d(in_channels = ngf*2, out_channels = ngf, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt5 = nn.ConvTranspose2d(in_channels = ngf, out_channels = 3, kernel_size=4, stride = 2, padding = 1, bias = False)

    def forward(self, t):
        t = self.convt1(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt2(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt3(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt4(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt5(t)
        t = F.tanh(t)

        return t

#---------------------------------------------------#
#implement the discriminator network

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = ndf, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv2 = nn.Conv2d(in_channels = ndf, out_channels = ndf*2, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv3 = nn.Conv2d(in_channels = ndf*2, out_channels = ndf*4, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv4 = nn.Conv2d(in_channels = ndf*4, out_channels = ndf*8, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv5 = nn.Conv2d(in_channels = ndf*8, out_channels = 1, kernel_size = 4, stride = 1, padding = 0, bias = False)

    def forward(self, t):
        t = self.conv1(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv2(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv3(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv4(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv5(t)
        t = F.sigmoid(t)

        return t

#---------------------------------------------------#
#create the instances of networks

generator = Generator()
discriminator = Discriminator()

#apply the weights_init function to randomly initialize weights to mean = 0 and std = 0.02
generator.apply(weights_init)
discriminator.apply(weights_init)

print(generator)
print(discriminator)

#---------------------------------------------------#
#define the loss function
criterion = nn.BCELoss()

#fixed noise
noise = torch.randn(64, nz, 1, 1).cuda()

#conventions for fake and real labels
real_label = 1
fake_label = 0

#create the optimizer instances
optimizer_d = optim.Adam(discriminator.parameters(), lr = lr, betas = (beta1, 0.999))
optimizer_g = optim.Adam(generator.parameters(), lr = lr, betas = (beta1, 0.999))

#---------------------------------------------------#
if torch.cuda.is_available():
    generator = generator.cuda()
    discriminator = discriminator.cuda()
    criterion = criterion.cuda()

Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor
#---------------------------------------------------#

#Training loop
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_data):

        #ones is passed when the data is coming from original dataset
        #zeros is passed when the data is coming from generator
        ones = Tensor(images.size(0), 1).fill_(1.0)
        zeros = Tensor(images.size(0),1).fill_(0.0)

        real_images = images.cuda()

        optimizer_g.zero_grad()

        #following is the input to the generator
        #we create tensor with random noise of size 100
        gen_input = np.random.normal(0,3,(512,100,4,4))
        gen_input = torch.tensor(gen_input, dtype = torch.float32)
        gen_input = gen_input.cuda()
        #we then pass it to generator()
        gen = generator(gen_input) #this returns a image

        #now calculate the loss wrt to discriminator output
        g_loss = criterion(discriminator(gen), ones)

        #backpropagation
        g_loss.backward()
        #update weights
        optimizer_g.step()

        #above was for generator network

        #now for the discriminator network
        optimizer_d.zero_grad()

        #calculate the real loss
        real_loss = criterion(discriminator(real_images), ones)
        #calculate the fake loss from the generated image
        fake_loss = criterion(discriminator(gen.detach()),zeros)
        #average out the losses
        d_loss = (real_loss + fake_loss)/2

        #backpropagation
        d_loss.backward()
        #update weights
        optimizer_d.step()

        if i%100 == 0:
            print("[EPOCH %d/%d] [Batch %d/%d] [D loss: %f] [G loss: %f]"%(epoch, epochs, i, len(dataset), d_loss.item(), g_loss.item()))

        total_batch = epoch * len(dataset) + i
        if total_batch%20 == 0:
            save_image(gen.data[:5], '/content/gdrive/My Drive/tttt/%d.png' % total_batch, nrow=5)

And here's the error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-0af32f223344> in <module>()
     18         gen_input = gen_input.cuda()
     19         #we then pass it to generator()
---> 20         gen = generator(gen_input) #this returns a image
     21
     22         #now calculate the loss wrt to discriminator output

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/batchnorm.py in __init__(self, num_features, eps, momentum, affine, track_running_stats)
     40         self.track_running_stats = track_running_stats
     41         if self.affine:
---> 42             self.weight = Parameter(torch.Tensor(num_features))
     43             self.bias = Parameter(torch.Tensor(num_features))
     44         else:

TypeError: expected CPU (got CUDA)

Any help would be appreciated. Thank you!

Output:
Do you use colab? Then you should activate the GPU. But if you want to stay on the CPU:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

Now do this on EVERY model or tensor you create, for example:
x = torch.tensor(...).to(device=device)
model = Model(...).to(device=device)

Then, if you switch around between cpu and gpu it handles it automaticaly for you. But as I said, you probably want to activate cuda by switching to colabs GPU


Instruction: Difficulties calculating mean square error between 2 tensors

Input:
I'm trying to build a loss function which will calculate the mean squared error of 2 tenors of the same size.
In other words I need a function that calcualtes the difference of every 2 cells (with the same row and column) cell on matrix A and matrix B, squares it and calculate the mean of the differences.
As far as I understand nn.MSELoss should do exactly that.
When I pass the 2 tensors to nn.MSELoss I'm getting the following error message:
RuntimeError: Boolean value of Tensor with more than one value is ambiguous

Here's my code
nn.MSELoss(stack_3[0,:],stack_7[0,:])

The tensors are floats of same shape.
stack_3.shape, stack_7.shape
(torch.Size([6131, 784]), torch.Size([6131, 784]))

Output:
nn.MSELoss is a callable class, not a function. You need to first define an instance of nn.MSELoss, then you can call it. Alternatively you can directly use torch.nn.functional.mse_loss.
from torch import nn
criterion = nn.MSELoss()
loss = criterion(stack_3[0, :], stack_7[0, :])

or
import torch.nn.functional as F
loss = F.mse_loss(stack_3[0, :], stack_7[0, :])


Instruction: Pytorch: Overfitting on a small batch: Debugging

Input:
I am building a multi-class image classifier.
There is a debugging trick to overfit on a single batch to check if there any deeper bugs in the program.
How to design the code in a way that can do it in a much portable format?
One arduous and a not smart way is to build a holdout train/test folder for a small batch where test class consists of 2 distribution - seen data and unseen data and if the model is performing better on seen data and poorly on unseen data, then we can conclude that our network doesn't have any deeper structural bug.
But, this does not seems like a smart and a portable way, and have to do it with every problem.
Currently, I have a dataset class where I am partitioning the data in train/dev/test in the below way -
def split_equal_into_val_test(csv_file=None, stratify_colname='y',
                              frac_train=0.6, frac_val=0.15, frac_test=0.25,
                              ):
    """
    Split a Pandas dataframe into three subsets (train, val, and test).

    Following fractional ratios provided by the user, where val and
    test set have the same number of each classes while train set have
    the remaining number of left classes
    Parameters
    ----------
    csv_file : Input data csv file to be passed
    stratify_colname : str
        The name of the column that will be used for stratification. Usually
        this column would be for the label.
    frac_train : float
    frac_val   : float
    frac_test  : float
        The ratios with which the dataframe will be split into train, val, and
        test data. The values should be expressed as float fractions and should
        sum to 1.0.
    random_state : int, None, or RandomStateInstance
        Value to be passed to train_test_split().

    Returns
    -------
    df_train, df_val, df_test :
        Dataframes containing the three splits.

    """
    df = pd.read_csv(csv_file).iloc[:, 1:]

    if frac_train + frac_val + frac_test != 1.0:
        raise ValueError('fractions %f, %f, %f do not add up to 1.0' %
                         (frac_train, frac_val, frac_test))

    if stratify_colname not in df.columns:
        raise ValueError('%s is not a column in the dataframe' %
                         (stratify_colname))

    df_input = df

    no_of_classes = 4
    sfact = int((0.1*len(df))/no_of_classes)

    # Shuffling the data frame
    df_input = df_input.sample(frac=1)

    df_temp_1 = df_input[df_input['labels'] == 1][:sfact]
    df_temp_2 = df_input[df_input['labels'] == 2][:sfact]
    df_temp_3 = df_input[df_input['labels'] == 3][:sfact]
    df_temp_4 = df_input[df_input['labels'] == 4][:sfact]

    dev_test_df = pd.concat([df_temp_1, df_temp_2, df_temp_3, df_temp_4])
    dev_test_y = dev_test_df['labels']
    # Split the temp dataframe into val and test dataframes.
    df_val, df_test, dev_Y, test_Y = train_test_split(
        dev_test_df, dev_test_y,
        stratify=dev_test_y,
        test_size=0.5,
        )

    df_train = df[~df['img'].isin(dev_test_df['img'])]

    assert len(df_input) == len(df_train) + len(df_val) + len(df_test)

    return df_train, df_val, df_test

def train_val_to_ids(train, val, test, stratify_columns='labels'): # noqa
    """
    Convert the stratified dataset in the form of dictionary : partition['train] and labels.

    To generate the parallel code according to https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel
    Parameters
    -----------
    csv_file : Input data csv file to be passed
    stratify_columns : The label column

    Returns
    -----------
    partition, labels:
        partition dictionary containing train and validation ids and label dictionary containing ids and their labels # noqa

    """
    train_list, val_list, test_list = train['img'].to_list(), val['img'].to_list(), test['img'].to_list() # noqa
    partition = {"train_set": train_list,
                 "val_set": val_list,
                 }
    labels = dict(zip(train.img, train.labels))
    labels.update(dict(zip(val.img, val.labels)))
    return partition, labels

P.S - I know about the Pytorch lightning and know that they have an overfitting feature which can be used easily but I don't want to move to PyTorch lightning.

Output:
I don't know how portable it will be, but a trick that I use is to modify the __len__ function in the Dataset.
If I modified it from
def __len__(self):
    return len(self.data_list)

to
def __len__(self):
    return 20

It will only output the first 20 elements in the dataset (regardless of shuffle). You only need to change one line of code and the rest should work just fine so I think it's pretty neat.


Instruction: Lstm with different input sizes for each batch (pyTorch)

Input:
I am switching from TensorFlow to PyTorch and I having some troubles with my net.
I have made a Collator(for the DataLoader) that pads each tensor(originally sentence) in each batch into the Maslen of each batch.
so I have different input sizes per each batch.
my network consists of LSTM -> LSTM -> DENSE
my question is, how can I specify this variable input size to the LSTM?
I assume that in TensorFlow I would do Input((None,x)) bedsore the LSTM.
Thank you in advance

Output:
The input size of the LSTM is not how long a sample is. So lets say you have a batch with three samples: The first one has the length of 10, the second 12 and the third 15. So what you already did is pad them all with zeros so that all three have the size 15. But the next batch may have been padded to 16. Sure.
But this 15 is not the input size of the LSTM. The input size in the size of one element of a sample in the batch. And that should always be the same.
For example when you want to classify names:
the inputs are names, for example "joe", "mark", "lucas".
But what the LSTM takes as input are the characters. So "J" then "o" so on. So as input size you have to put how many dimensions one character have.
If you use embeddings, the embedding size. When you use one-hot encoding, the vector size (probably 26). An LSTM takes iteratively the characters of the words. Not the entire word at once.
self.lstm = nn.LSTM(input_size=embedding_size, ...)

I hope this answered your question, if not please clearify it! Good luck!


Instruction: How can I fix this expected CUDA got CPU error in PyTorch?

Input:
I've been struggling to find what's wrong in my code. I'm trying to implement DCGAN paper and from the past 2 days, I'm going through these errors. Could anyone please help me fix this?
I'm training this on Google colab with GPU runtime but I'm getting this error. Yesterday, I implemented the first GAN paper by Ian Goodfellow and I did not got this error. I don't know what's happening any help would be appreciated. Also, please check whether the gen_input is correct or not.
I already asked this question and no one replied to the old post. one person replied to it but all he said was to change the lines up and down which also gives the same error. please please help me
Here is the code:
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torchvision
import torchvision.transforms as transforms
from torchvision.utils import save_image
import torch.optim as optim

lr = 0.00002 #learning rate
nc = 3 #color channels
nz = 100 #size of latent vector or size of generator input
ngf = 64 #size of feature maps in generator
ndf = 64 #size of feature maps in discriminator
height = 128 #height of the image
width = 128 #width of the image
num_epochs = 5 #the variable name tells everything
workers = 2 #number of workers to load the data in batches
batch_size = 64 #batch size
image_size = 128 #resizing parameter
root = './simpsons/' #path to the training directory
beta1 = 0.5

img_shape = (nc, height, width)

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.convt1 = nn.ConvTranspose2d(in_channels = nz, out_channels = ngf*8, kernel_size = 4, stride = 1, padding = 0, bias = False)
        self.convt2 = nn.ConvTranspose2d(in_channels = ngf*8, out_channels = ngf*4, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt3 = nn.ConvTranspose2d(in_channels = ngf*4, out_channels = ngf*2, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt4 = nn.ConvTranspose2d(in_channels = ngf*2, out_channels = ngf, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt5 = nn.ConvTranspose2d(in_channels = ngf, out_channels = 3, kernel_size=4, stride = 2, padding = 1, bias = False)

    def forward(self, t):
        t = self.convt1(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt2(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt3(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt4(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt5(t)
        t = F.tanh(t)

        return t

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = ndf, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv2 = nn.Conv2d(in_channels = ndf, out_channels = ndf*2, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv3 = nn.Conv2d(in_channels = ndf*2, out_channels = ndf*4, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv4 = nn.Conv2d(in_channels = ndf*4, out_channels = ndf*8, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv5 = nn.Conv2d(in_channels = ndf*8, out_channels = 1, kernel_size = 4, stride = 1, padding = 0, bias = False)

    def forward(self, t):
        t = self.conv1(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv2(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv3(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv4(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv5(t)
        t = F.sigmoid(t)

        return t

def weights_init(m):
    classname = m.__class__.__name__ #returns the class name(eg: Conv2d or ConvTranspose2d)
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02) #0.0 is mean and 0.02 is standard deviation
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1, 0.02) #1 is mean and 0.02 is standard deviation
        nn.init.constant_(m.bias.data, 0.0)

def load_data(image_size, root):
    transform = transforms.Compose([
        transforms.Resize(image_size),
        transforms.ToTensor(),
        transforms.Normalize((0.486, 0.486, 0.486), (0.486, 0.486, 0.486))
        ])

    train_set = torchvision.datasets.ImageFolder(root = root, transform = transform)

    return train_set

#getting the batches of data
train_set = load_data(image_size, root)
dataloader = torch.utils.data.DataLoader(train_set, batch_size = batch_size, shuffle = True, num_workers = workers)

generator = Generator()
discriminator = Discriminator()

generator.apply(weights_init)
discriminator.apply(weights_init)

print(generator)
print(discriminator)

criterion = nn.BCELoss()

noise = torch.randn(64, nz, 1, 1)

optimizer_G = optim.Adam(generator.parameters(), lr = lr, betas=(beta1, 0.999))
optimizer_D = optim.Adam(discriminator.parameters(), lr = lr, betas=(beta1, 0.999))

if torch.cuda.is_available():
    print("CUDA available")
    generator = generator.to('cuda')
    discriminator = discriminator.to('cuda')
    criterion = criterion.cuda('cuda')
    Tensor = torch.cuda.FloatTensor
    print("Networks moved on to cuda")

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(dataloader):

        val = Tensor(images.size(0), 1).fill_(1.0)
        fake = Tensor(images.size(0),1).fill_(0.0)

        real_images = images

        optimizer_G.zero_grad()

        gen_input = Tensor(np.random.normal(0,1,(512,100,4,4)))
        gen = generator(gen_input)

        g_loss = loss_func(discriminator(gen), val)

        g_loss.backward()
        optimizer_G.step()

        optimizer_D.zero_grad()

        real_loss = loss_func(discriminator(real_images), val)
        fake_loss = loss_func(discriminator(gen.detach()),fake)
        d_loss = (real_loss + fake_loss)/2

        d_loss.backward()
        optimizer_D.step()

        if i%900 == 0:
            print("[EPOCH %d/%d] [Batch %d/%d] [D loss: %f] [G loss: %f]"%(epoch, num_epochs, i, len(dataset), d_loss.item(), g_loss.item()))

        total_batch = epoch * len(dataset) + i
        if total_batch%400 == 0:
            save_image(gen.data[:25], 'output/%d.png' % total_batch, nrow=5)

And here's the error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-0af32f223344> in <module>()
     18         gen_input = gen_input.cuda()
     19         #we then pass it to generator()
---> 20         gen = generator(gen_input) #this returns a image
     21
     22         #now calculate the loss wrt to discriminator output

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/batchnorm.py in __init__(self, num_features, eps, momentum, affine, track_running_stats)
     40         self.track_running_stats = track_running_stats
     41         if self.affine:
---> 42             self.weight = Parameter(torch.Tensor(num_features))
     43             self.bias = Parameter(torch.Tensor(num_features))
     44         else:

TypeError: expected CPU (got CUDA)

Any help would be appreciated. Thank you!

Output:
You are using nn.BatchNorm2d in a wrong way.
BatchNorm is a layer,  just like Conv2d. It has internal parameters and buffers.
Therefore, you must define these layers in the __init__ of your generator/discriminator.
Right now, you define the layer in your forward pass - this is wrong in so many ways...


Instruction: Torch installed but repeat_interleave not found

Input:
I have PyTorch installed.
import torch runs without error. However, the function torch.repeat_interleave() is not found:
x = torch.tensor([1, 2, 3])
x.repeat_interleave(2)

gives AttributeError: 'Tensor' object has no attribute 'repeat_interleave'
Why?

Output: The torch.repeat_interleave operator was introduced in 1.1.0 release of pytorch, so please, consider updating to 1.1.0+ version of pytorch to use this method smoothly.


Instruction: Generic Computation of Distance Matrices in Pytorch

Input:
I have two tensors a & b of shape (m,n), and I would like to compute a distance matrix m using some distance metric d. That is, I want m[i][j] = d(a[i], b[j]). This is somewhat like cdist(a,b) but assuming a generic distance function d which is not necessarily a p-norm distance. Is there a generic way to implement this in PyTorch?
And a more specific side question: Is there an efficient way to perform this with the following metric
d(x,y) = 1 - cos(x,y)

edit
I've solved the specific case above using this answer:
def metric(a, b, eps=1e-8):
    a_norm, b_norm = a.norm(dim=1)[:, None], b.norm(dim=1)[:, None]
    a_norm = a / torch.max(a_norm, eps * torch.ones_like(a_norm))
    b_norm = b / torch.max(b_norm, eps * torch.ones_like(b_norm))
    similarity_matrix = torch.mm(a_norm, b_norm.transpose(0, 1))
    return 1 - similarity_matrix

Output:
I'd suggest using broadcasting: since a,b both have shape (m,n) you can compute
m = d( a[None, :, :], b[:, None, :])

where d needs to operate on the last dimension, so for instance
def d(a,b): return 1 - (a * b).sum(dim=2) / a.pow(2).sum(dim=2).sqrt() / b.pow(2).sum(dim=2).sqrt()

(here I assume that cos(x,y) represents the normalized inner product between x and y)


Instruction: Add dense layer on top of Huggingface BERT model

Input:
I want to add a dense layer on top of the bare BERT Model transformer outputting raw hidden-states, and then fine tune the resulting model. Specifically, I am using this base model. This is what the model should do:

Encode the sentence (a vector with 768 elements for each token of the sentence)
Keep only the first vector (related to the first token)
Add a dense layer on top of this vector, to get the desired transformation

So far, I have successfully encoded the sentences:
from sklearn.neural_network import MLPRegressor

import torch

from transformers import AutoModel, AutoTokenizer

# List of strings
sentences = [...]
# List of numbers
labels = [...]

tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")

# 2D array, one line per sentence containing the embedding of the first token
encoded_sentences = torch.stack([model(**tokenizer(s, return_tensors='pt'))[0][0][0]
                                 for s in sentences]).detach().numpy()

regr = MLPRegressor()
regr.fit(encoded_sentences, labels)

In this way I can train a neural network by feeding it with the encoded sentences. However, this approach clearly does not fine tune the base BERT model. Can anybody help me? How can I build a model (possibly in pytorch or using the Huggingface library) that can be entirely fine tuned?

Output:
There are two ways to do it: Since you are looking to fine-tune the model for a downstream task similar to classification, you can directly use:
BertForSequenceClassification class. Performs fine-tuning of logistic regression layer on the output dimension of 768.
Alternatively, you can define a custom module, that created a bert model based on the pre-trained weights and adds layers on top of it.
from transformers import BertModel
class CustomBERTModel(nn.Module):
    def __init__(self):
          super(CustomBERTModel, self).__init__()
          self.bert = BertModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
          ### New layers:
          self.linear1 = nn.Linear(768, 256)
          self.linear2 = nn.Linear(256, 3) ## 3 is the number of classes in this example

    def forward(self, ids, mask):
          sequence_output, pooled_output = self.bert(
               ids,
               attention_mask=mask)

          # sequence_output has the following shape: (batch_size, sequence_length, 768)
          linear1_output = self.linear1(sequence_output[:,0,:].view(-1,768)) ## extract the 1st token's embeddings

          linear2_output = self.linear2(linear2_output)

          return linear2_output

tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
model = CustomBERTModel() # You can pass the parameters if required to have more flexible model
model.to(torch.device("cpu")) ## can be gpu
criterion = nn.CrossEntropyLoss() ## If required define your own criterion
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()))

for epoch in epochs:
    for batch in data_loader: ## If you have a DataLoader()  object to get the data.

        data = batch[0]
        targets = batch[1] ## assuming that data loader returns a tuple of data and its targets

        optimizer.zero_grad()
        encoding = tokenizer.batch_encode_plus(data, return_tensors='pt', padding=True, truncation=True,max_length=50, add_special_tokens = True)
        outputs = model(input_ids, attention_mask=attention_mask)
        outputs = F.log_softmax(outputs, dim=1)
        input_ids = encoding['input_ids']
        attention_mask = encoding['attention_mask']
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()


Instruction: PyTorch Factorial Function

Input:
There does not seem to be a PyTorch function for computing a factorial. Is there a method to do this in PyTorch? I am looking to manually compute a Poisson distribution in Torch (I am aware this exists: https://pytorch.org/docs/stable/generated/torch.poisson.html) and the formula requires a factorial in the denominator.
Poisson Distribution: https://en.wikipedia.org/wiki/Poisson_distribution

Output:
I think you can find it as torch.jit._builtins.math.factorial BUT pytorch as well as numpy and scipy (Factorial in numpy and scipy) uses python's builtin math.factorial:
import math

import numpy as np
import scipy as sp
import torch

print(torch.jit._builtins.math.factorial is math.factorial)
print(np.math.factorial is math.factorial)
print(sp.math.factorial is math.factorial)

True
True
True

But, in contrast, scipy in addition to "mainstream" math.factorial contains the very "special" factorial function scipy.special.factorial. Unlike function from math module it operates on arrays:
from scipy import special

print(special.factorial is math.factorial)

False

# the all known factorial functions
factorials = (
    math.factorial,
    torch.jit._builtins.math.factorial,
    np.math.factorial,
    sp.math.factorial,
    special.factorial,
)

# Let's run some tests
tnsr = torch.tensor(3)

for fn in factorials:
    try:
        out = fn(tnsr)
    except Exception as err:
        print(fn.__name__, fn.__module__, ':', err)
    else:
        print(fn.__name__, fn.__module__, ':', out)

factorial math : 6
factorial math : 6
factorial math : 6
factorial math : 6
factorial scipy.special._basic : tensor(6., dtype=torch.float64)

tnsr = torch.tensor([1, 2, 3])

for fn in factorials:
    try:
        out = fn(tnsr)
    except Exception as err:
        print(fn.__name__, fn.__module__, ':', err)
    else:
        print(fn.__name__, fn.__module__, ':', out)

factorial math : only integer tensors of a single element can be converted to an index
factorial math : only integer tensors of a single element can be converted to an index
factorial math : only integer tensors of a single element can be converted to an index
factorial math : only integer tensors of a single element can be converted to an index
factorial scipy.special._basic : tensor([1., 2., 6.], dtype=torch.float64)


Instruction: How to randomly set a fixed number of elements in each row of a tensor in PyTorch

Input:
I was wondering if there is any more efficient alternative for the below code, without using the "for" loop in the 4th line?
import torch
n, d = 37700, 7842
k = 4
sample = torch.cat([torch.randperm(d)[:k] for _ in range(n)]).view(n, k)
mask = torch.zeros(n, d, dtype=torch.bool)
mask.scatter_(dim=1, index=sample, value=True)

Basically, what I am trying to do is to create an n by d mask tensor, such that in each row exactly k random elements are True.

Output:
Here's a way to do this with no loop. Let's start with a random matrix where all elements are drawn iid, in this case uniformly on [0,1]. Then we take the k'th quantile for each row and set all smaller or equal elements to True and the rest to False on each row:
rand_mat = torch.rand(n, d)
k_th_quant = torch.topk(rand_mat, k, largest = False)[0][:,-1:]
mask = rand_mat <= k_th_quant

No loop needed :) x2.1598 faster than the code you attached on my CPU.


Instruction: PyTorch installation problem- package not found using Jupyter notebook and Conda navigator

Input:
I have tried to install PyTorch using the installation code form official PyTorch website.
I run it locally in the Jupyter notebook on Conda navigator
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

I received the following mistake
    PackagesNotFoundError: The following packages are not available from current channels:

  - pytorch
  - cudatoolkit=10.2

Current channels:

  - https://conda.anaconda.org/pytorch/win-32
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/win-32
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-32
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-32
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

What to do?

Output:
TL;DR Use 64-bit Anaconda
conda manages packages for one platform and architecture.
It looks like you installed 32-Bit (x86) Win Anaconda version, here:
- https://conda.anaconda.org/pytorch/win-32

You can check that channel under that link (https://conda.anaconda.org/pytorch/win-32) does not contain any pytorch package, and same is for this one: https://conda.anaconda.org/pytorch/noarch
If you look at the win64: https://conda.anaconda.org/pytorch/win-64 it actually contains pytorch packages.
So, there is no pytorch x86 packages in pytorch channels and in addition it is not possible to create environment of another architecture, meaning that you need to install 64-bit Anaconda to use pytorch.


Instruction: IndexError: Target 60972032 is out of bounds

Input:
I am trying to call cross entropy loss but it says the index is out of range
loss = nn.CrossEntropyLoss()
target = torch.empty(128, dtype=torch.long)
result = loss(output, target)

Note the output has the shape torch.Size([128, 10])

Output:
The target tensor from provided example is not initialized, see torch.empty
It's empty, to fix that use, for example, .random_ method, like in CrossEntropyLoss docs example:
...
target = torch.empty(128, dtype=torch.long).random_(10)
...


Instruction: Modify existing Pytorch code to run on multiple GPUs

Input:
I'm trying to run Pytoch UNet from the following link on 2 or more GPUs
Pytorch-UNet github
the changes the I did till now is:
1.
from:
net = UNet(n_channels=3, n_classes=1, bilinear=True)
logging.info(f'Network:\n'
             f'\t{net.module.n_channels} input channels\n'
             f'\t{net.module.n_classes} output channels (classes)\n'
             f'\t{"Bilinear" if net.module.bilinear else "Transposed conv"} upscaling')

to:
net = UNet(n_channels=3, n_classes=1, bilinear=True)
net = nn.DataParallel(net)
logging.info(f'Network:\n'
             f'\t{net.module.n_channels} input channels\n'
             f'\t{net.module.n_classes} output channels (classes)\n'
             f'\t{"Bilinear" if net.module.bilinear else "Transposed conv"} upscaling')

in each place where was:
net.<something>

replaced to:
net.module.<something>

I know that pytorch see more that 1 GPU because torch.cuda.device_count() return

2

.
But as long a I try to run train that need more momery than what the first GPU have I'm getting:

RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0;
11.91 GiB total capacity; 10.51 GiB already allocated; 82.56 MiB free; 818.92 MiB cached)

I change required memory to train by changing the batch size.
Any help welcome
EDIT
I see that training run twice faster with 2 GPUs, but max batch size of run with single GPU is the same as for two GPU. Is there any way to use the memory of 2 GPUs together during a single training run?

Output:
My mistake was changing output = net(input) (commonly named as model) to:
output = net.module(input)
you can find information here


Instruction: Extracting blocks from block diagonal PyTorch tensor

Input:
I have a tensor of shape (m*n, m*n) and I want to extract a tensor of size (n, m*n) containing the m blocks of size n*n that are on the diagonal. For example:
>>> a
tensor([[1, 2, 0, 0],
        [3, 4, 0, 0],
        [0, 0, 5, 6],
        [0, 0, 7, 8]])

I want to have a function extract(a, m, n) that will output:
>>> extract(a, 2, 2)
tensor([[1, 2, 5, 6],
        [3, 4, 7, 8]])

I've thought of using some kind of slicing, because the blocks can be expressed by:
>>> for i in range(m):
...     print(a[i*m: i*m + n, i*m: i*m + n])
tensor([[1, 2],
        [3, 4]])
tensor([[5, 6],
        [7, 8]])

Output:
You can take advantage of reshape and slicing:
import torch
import numpy as np

def extract(a, m, n):
  s=(range(m), np.s_[:], range(m), np.s_[:])  # the slices of the blocks
  a.reshape(m, n, m, n)[s]  # reshaping according to blocks and slicing
  return a.reshape(m*n, n).T  # reshape to desired output format

Example:
a = torch.arange(36).reshape(6,6)
a
tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11],
        [12, 13, 14, 15, 16, 17],
        [18, 19, 20, 21, 22, 23],
        [24, 25, 26, 27, 28, 29],
        [30, 31, 32, 33, 34, 35]])

extract(a, 3, 2)

tensor([[ 0,  6, 14, 20, 28, 34],
        [ 1,  7, 15, 21, 29, 35]])

extract(a, 2, 3)

tensor([[ 0,  6, 12, 21, 27, 33],
        [ 1,  7, 13, 22, 28, 34],
        [ 2,  8, 14, 23, 29, 35]])


Instruction: How can I create a 2D tensor with one element a sine and the other a cosine wave?

Input:
I'm doing
train_data = np.array([np.sin(time), np.cos(time)])

This gives me something with a shape of (2, 4000). I think I need it to be (4000, 2), so for each timestep, I can get the sin and cos.
How can I do that?

Output:
You can use numpy transpose to switch the array around.
Try this:
train_data = np.array([np.sin(time), np.cos(time)]).T

Here's what I did to get (4000,2).
import numpy as np
time = np.arange(4000)
train_data = np.array([np.sin(time), np.cos(time)])
print (train_data.shape)
print (train_data.T.shape)

The output for this was:
(2, 4000)
(4000, 2)


Instruction: How do I rotate a PyTorch image tensor around it's center in a way that supports autograd?

Input:
I'd like to randomly rotate an image tensor (B, C, H, W) around it's center (2d rotation I think?). I would like to avoid using NumPy and Kornia, so that I basically only need to import from the torch module. I'm also not using torchvision.transforms, because I need it to be autograd compatible. Essentially I'm trying to create an autograd compatible version of torchvision.transforms.RandomRotation() for visualization techniques like DeepDream (so I need to avoid artifacts as much as possible).
import torch
import math
import random
import torchvision.transforms as transforms
from PIL import Image

# Load image
def preprocess_simple(image_name, image_size):
    Loader = transforms.Compose([transforms.Resize(image_size), transforms.ToTensor()])
    image = Image.open(image_name).convert('RGB')
    return Loader(image).unsqueeze(0)

# Save image
def deprocess_simple(output_tensor, output_name):
    output_tensor.clamp_(0, 1)
    Image2PIL = transforms.ToPILImage()
    image = Image2PIL(output_tensor.squeeze(0))
    image.save(output_name)

# Somehow rotate tensor around it's center
def rotate_tensor(tensor, radians):
    ...
    return rotated_tensor

# Get a random angle within a specified range
r_degrees = 5
angle_range = list(range(-r_degrees, r_degrees))
n = random.randint(angle_range[0], angle_range[len(angle_range)-1])

# Convert angle from degrees to radians
ang_rad = angle * math.pi / 180

# test_tensor = preprocess_simple('path/to/file', (512,512))
test_tensor = torch.randn(1,3,512,512)

# Rotate input tensor somehow
output_tensor = rotate_tensor(test_tensor, ang_rad)

# Optionally use this to check rotated image
# deprocess_simple(output_tensor, 'rotated_image.jpg')

Some example outputs of what I'm trying to accomplish:

Output:
So the grid generator and the sampler are sub-modules of the Spatial Transformer (JADERBERG, Max, et al.). These sub-modules are not trainable, they let you apply a learnable, as well as non-learnable, spatial transformation.
Here I take these two submodules and use them to rotate an image by theta using PyTorch's functions torch.nn.functional.affine_grid and torch.nn.functional.affine_sample (these functions are implementations of the generator and the sampler, respectively):
import torch
import torch.nn.functional as F
import numpy as np
import matplotlib.pyplot as plt

def get_rot_mat(theta):
    theta = torch.tensor(theta)
    return torch.tensor([[torch.cos(theta), -torch.sin(theta), 0],
                         [torch.sin(theta), torch.cos(theta), 0]])

def rot_img(x, theta, dtype):
    rot_mat = get_rot_mat(theta)[None, ...].type(dtype).repeat(x.shape[0],1,1)
    grid = F.affine_grid(rot_mat, x.size()).type(dtype)
    x = F.grid_sample(x, grid)
    return x

#Test:
dtype =  torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor
#im should be a 4D tensor of shape B x C x H x W with type dtype, range [0,255]:
plt.imshow(im.squeeze(0).permute(1,2,0)/255) #To plot it im should be 1 x C x H x W
plt.figure()
#Rotation by np.pi/2 with autograd support:
rotated_im = rot_img(im, np.pi/2, dtype) # Rotate image by 90 degrees.
plt.imshow(rotated_im.squeeze(0).permute(1,2,0)/255)

In the example above, assume we take our image, im, to be a dancing cat in a skirt:

rotated_im will be a 90-degrees CCW rotated dancing cat in a skirt:

And this is what we get if we call rot_img with theta eqauls to np.pi/4:

And the best part that it's differentiable w.r.t the input and has autograd support! Hooray!


Instruction: Pytorch Transform library explaination

Input:
transforms.Normalize([0.5]*3, [0.5]*3)
Can someone help me to understand what this and how it works?

Output:
You have the documentation for the Normalizetransform here. It says : Normalize a tensor image with mean and standard deviation. Given mean: (mean[1],...,mean[n]) and std: (std[1],..,std[n]) for n channels, this transform will normalize each channel of the input torch.Tensor i.e., output[channel] = (input[channel] - mean[channel]) / std[channel]
So in your case, you are constructing a Normalize transform with mean=std=[0.5,0.5,0.5]. This means that you are expecting an input with 3 channels, and for each channel you want to normalize with the function
x -> (x-0.5)/0.5 = 2x-1


Instruction: how does nn.embedding for developing an encoder-decoder model works?

Input:
In this tutorial, it teaches how to develop a simple encoder-decoder model with attention using pytorch.
However, in the encoder or decoder, self.embedding = nn.Embedding(input_size, hidden_size) (or similar) is defined. In pytorch documents, nn.Embedding is defined as "A simple lookup table that stores embeddings of a fixed dictionary and size."
So I am confused that, in the initialization, where does this lookup table has come from? Does it initialize some random embeddings for the indices and then they will be trained? Is it really necessary to be in the encoder/decoder part?
Thanks in advance.

Output:
Answering the last bit first: Yes, we do need Embedding or an equivalent. At least when dealing with discrete inputs (e.g. letters or words of a language),  because these tokens come encoded as integers (e.g. 'a' -> 1, 'b' -> 2, etc.), but those numbers do not carry meaning: The letter 'b' is not "like 'a', but more", which its original encoding would suggest. So we provide the Embedding so that the network can learn how to represent these letters by something useful, e.g. making vowels similar to one another in some way.
During the initialization, the embedding vector are sampled randomly, in the same fashion as other weights in the model, and also get optimized with the rest of the model. It is also possible to initialize them from some pretrained embeddings (e.g. from word2vec, Glove, FastText), but caution must then be exercised not to destroy them by backprop through randomly initialized model.

Embeddings are not stricly necessary, but it would be very wasteful to force network to learn that 13314 ('items') is very similar to 89137 ('values'), but completely different to 13315 ('japan'). And it would probably not even remotely converge anyway.


Instruction: What can I use in Pytorch to remplace caffe's weight filler

Input:
I'm writing a pytorch model based on the caffe model below. Do you know how I can write weight filler and bias filler in pytorch ?
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "pool2"
  top: "conv3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  convolution_param {
    num_output: 128
    pad_h: 0
    pad_w: 1
    kernel_h: 1
    kernel_w: 3
    stride: 1
    weight_filler {
      type: "gaussian"
      std: 0.1
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}

Thank you

Output:
Pytorch has torch.nn.init library to help with init weights of a network.
You probably want to use nn.init.normal_ for the "gaussian" filler, and nn.init.constant_ for the "constant" filler of the bias.
You can use a function to fill the weights of a module m:
def init_weights(m):
    if type(m) == nn.Conv2d:
        torch.nn.init.normal_(m.weight, std=0.1)
        if m.bias is not None:
            torch.nn.init.constant_(m.bias, val=0)

# define the net
net = MyCaffeLikeNetwork()
# use the function to init all weights of the net
net.apply(init_weights)

For more information on weight init in pytorch you can look at this detailed answer.


Instruction: PyTorch - RuntimeError: [enforce fail at inline_container.cc:209] . file not found: archive/data.pkl

Input:
Problem
I'm trying to load a file using PyTorch, but the error states archive/data.pkl does not exist.
Code
import torch
cachefile = 'cacheddata.pth'
torch.load(cachefile)

Output
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-8edf1f27a4bd> in <module>
      1 import torch
      2 cachefile = 'cacheddata.pth'
----> 3 torch.load(cachefile)

~/opt/anaconda3/envs/matching/lib/python3.8/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    582                     opened_file.seek(orig_position)
    583                     return torch.jit.load(opened_file)
--> 584                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
    585         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
    586

~/opt/anaconda3/envs/matching/lib/python3.8/site-packages/torch/serialization.py in _load(zip_file, map_location, pickle_module, **pickle_load_args)
    837
    838     # Load the data (which may in turn use `persistent_load` to load tensors)
--> 839     data_file = io.BytesIO(zip_file.get_record('data.pkl'))
    840     unpickler = pickle_module.Unpickler(data_file, **pickle_load_args)
    841     unpickler.persistent_load = persistent_load

RuntimeError: [enforce fail at inline_container.cc:209] . file not found: archive/data.pkl

Hypothesis
I'm guessing this has something to do with pickle, from the docs:

This save/load process uses the most intuitive syntax and involves the
least amount of code. Saving a model in this way will save the entire
module using Python’s pickle module. The disadvantage of this approach
is that the serialized data is bound to the specific classes and the
exact directory structure used when the model is saved. The reason for
this is because pickle does not save the model class itself. Rather,
it saves a path to the file containing the class, which is used during
load time. Because of this, your code can break in various ways when
used in other projects or after refactors.

Versions

PyTorch version: 1.6.0
Python version: 3.8.0

Output: Turned out the file was somehow corrupted. After generating it again it loaded without issue.


Instruction: Pytorch model doesn't learn identity function?

Input:
I wrote some models in pytorch which was not able to learn anything even after many epochs. In order to debug the problem I made a simple model which models identity function of an input. The difficulty is this model also doesn't learn nothing despite training for 50k epochs,
import torch
import torch.nn as nn

torch.manual_seed(1)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.input = nn.Linear(2,4)
        self.hidden = nn.Linear(4,4)
        self.output = nn.Linear(4,2)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        self.dropout = nn.Dropout(0.5)
    def forward(self,x):
        x = self.input(x)
        x = self.dropout(x)
        x = self.relu(x)
        x = self.hidden(x)
        x = self.dropout(x)
        x = self.relu(x)
        x = self.output(x)
        x = self.softmax(x)
        return x

X = torch.tensor([[1,0],[1,0],[0,1],[0,1]],dtype=torch.float)

net = Net()

criterion = nn.CrossEntropyLoss()

opt = torch.optim.Adam(net.parameters(), lr=0.001)

for i in range(100000):
    opt.zero_grad()
    y = net(X)
    loss = criterion(y,torch.argmax(X,dim=1))
    loss.backward()
    if i%500 ==0:
        print("Epoch: ",i)
        print(torch.argmax(y,dim=1).detach().numpy().tolist())
        print("Loss: ",loss.item())
        print()

Output
Epoch:  52500
[0, 0, 1, 0]
Loss:  0.6554909944534302

Epoch:  53000
[0, 0, 0, 0]
Loss:  0.7004914283752441

Epoch:  53500
[0, 0, 0, 0]
Loss:  0.7156486511230469

Epoch:  54000
[0, 0, 0, 0]
Loss:  0.7171240448951721

Epoch:  54500
[0, 0, 0, 0]
Loss:  0.691678524017334

Epoch:  55000
[0, 0, 0, 0]
Loss:  0.7301554679870605

Epoch:  55500
[0, 0, 0, 0]
Loss:  0.728650689125061

What is wrong with my implementation?

Output:
There are a few mistakes:

Missing optimizer.step():

optimizer.step() updates the parameters based on backpropagated gradients and other accumulated momentum and all.

Usage of softmax with CrossEntropy Loss:

Pytorch CrossEntropyLoss  criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class. i.e. it applies softmax then takes negative log. So in your case you are taking softmax(softmax(output)). Correct way is use linear output layer while training and use softmax layer or just take argmax for prediction.

High dropout value for small network:

Which results in underfitting.
Here's the corrected code:
import torch
import torch.nn as nn

torch.manual_seed(1)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.input = nn.Linear(2,4)
        self.hidden = nn.Linear(4,4)
        self.output = nn.Linear(4,2)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        # self.dropout = nn.Dropout(0.0)
    def forward(self,x):
        x = self.input(x)
        # x = self.dropout(x)
        x = self.relu(x)
        x = self.hidden(x)
        # x = self.dropout(x)
        x = self.relu(x)
        x = self.output(x)
        # x = self.softmax(x)
        return x

    def predict(self, x):
        with torch.no_grad():
            out = self.forward(x)
        return self.softmax(out)

X = torch.tensor([[1,0],[1,0],[0,1],[0,1]],dtype=torch.float)

net = Net()

criterion = nn.CrossEntropyLoss()

opt = torch.optim.Adam(net.parameters(), lr=0.001)

for i in range(100000):
    opt.zero_grad()
    y = net(X)
    loss = criterion(y,torch.argmax(X,dim=1))
    loss.backward()
    # This was missing before
    opt.step()
    if i%500 ==0:
        print("Epoch: ",i)
        pred = net.predict(X)
        print(f'prediction: {torch.argmax(pred, dim=1).detach().numpy().tolist()}, actual: {torch.argmax(X,dim=1)}')
        print("Loss: ", loss.item())

Output:
Epoch:  0
prediction: [0, 0, 0, 0], actual: tensor([0, 0, 1, 1])
Loss:  0.7042869329452515
Epoch:  500
prediction: [0, 0, 1, 1], actual: tensor([0, 0, 1, 1])
Loss:  0.1166711300611496
Epoch:  1000
prediction: [0, 0, 1, 1], actual: tensor([0, 0, 1, 1])
Loss:  0.05215628445148468
Epoch:  1500
prediction: [0, 0, 1, 1], actual: tensor([0, 0, 1, 1])
Loss:  0.02993333339691162
Epoch:  2000
prediction: [0, 0, 1, 1], actual: tensor([0, 0, 1, 1])
Loss:  0.01916157826781273
Epoch:  2500
prediction: [0, 0, 1, 1], actual: tensor([0, 0, 1, 1])
Loss:  0.01306679006665945
Epoch:  3000
prediction: [0, 0, 1, 1], actual: tensor([0, 0, 1, 1])
Loss:  0.009280549362301826
.
.
.


Instruction: Is there a training/validation split happening internally, or is there just one training set and testing set?

Input:
so recently I've been following the tutorial in https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html and I came up with the following question: is there a training/validation split happening internally?
The thing is, in this tutorial, the main dataset is spliced into training and testing. Here, the training set is used for training and the testing in the evaluate() function.
To my knowledge, when dealing with neural networks usually the data is split into 3 sets: training, validation and testing. In this tutorial though, it is only split into training and testing. For what I know, usually the model is trained and then evaluated, and the weights are then updated according to what was learnt in the evaluation step. However, I can't seem to find any connection between the evaluate function and training. Therefore, in this example the model is being evaluated AND tested using the same dataset.
Is there something here that I might be missing? Is there an internal split of the training dataset happening during training (into training and validation) and the function evaluate() is simply used for testing the performance of the model?
for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)```

Output:
Is there a training/validation split happening internally?
Is there an internal split of the training dataset happening during
training (into training and validation) and the function evaluate() is
simply used for testing the performance of the model?

No, you are not missing anything. What you see is exactly what's being done there. There is no internal splitting happening. It's just an example to show how something is done in pytorch without cluttering it unncessarily.
Some datasets such as CIFAR10/CIFAR100, come only with a train/test set and usually it has been the norm to just train and then evaluate on the test set in examples. However, nothing stops you from splitting the training set however you like, it's up to you. In such tutorials they just tried to keep everything as simple as possible.


Instruction: what's the difference between "self-attention mechanism" and "full-connection" layer?

Input: I am confused with these two structures. In theory, the output of them are all connected to their input. what magic make 'self-attention mechanism' is more powerful than the full-connection layer?

Output:
Ignoring details like normalization, biases, and such, fully connected networks are fixed-weights:
f(x) = (Wx)

where W is learned in training, and fixed in inference.
Self-attention layers are dynamic, changing the weight as it goes:
attn(x) = (Wx)
f(x) = (attn(x) * x)

Again this is ignoring a lot of details but there are many different implementations for different applications and you should really check a paper for that.


Instruction: What is the idea behind using nn.Identity for residual learning?

Input:
So, I've read about half the original ResNet paper, and am trying to figure out how to make my version for tabular data.
I've read a few blog posts on how it works in PyTorch, and I see heavy use of nn.Identity(). Now, the paper also frequently uses the term identity mapping. However, it just refers to adding the input for a stack of layers the output of that same stack in an element-wise fashion. If the in and out dimensions are different, then the paper talks about padding the input with zeros or using a matrix W_s to project the input to a different dimension.
Here is an abstraction of a residual block I found in a blog post:

class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, activation='relu'):
        super().__init__()
        self.in_channels, self.out_channels, self.activation = in_channels, out_channels, activation
        self.blocks = nn.Identity()
        self.shortcut = nn.Identity()

    def forward(self, x):
        residual = x
        if self.should_apply_shortcut: residual = self.shortcut(x)
        x = self.blocks(x)
        x += residual
        return x

    @property
    def should_apply_shortcut(self):
        return self.in_channels != self.out_channels

block1 = ResidualBlock(4, 4)

And my own application to a dummy tensor:
x = tensor([1, 1, 2, 2])
block1 = ResidualBlock(4, 4)
block2 = ResidualBlock(4, 6)
x = block1(x)
print(x)
x = block2(x)
print(x)

>>> tensor([2, 2, 4, 4])
>>> tensor([4, 4, 8, 8])

So at the end of it, x = nn.Identity(x) and I'm not sure the point of its use except to mimic math lingo found in the original paper. I'm sure that's not the case though, and that it has some hidden use that I'm just not seeing yet. What could it be?
EDIT Here is another example of implementing residual learning, this time in Keras. It does just what I suggested above and just keeps a copy of the input for adding to the output:
def residual_block(x: Tensor, downsample: bool, filters: int,                                        kernel_size: int = 3) -> Tensor:
    y = Conv2D(kernel_size=kernel_size,
               strides= (1 if not downsample else 2),
               filters=filters,
               padding="same")(x)
    y = relu_bn(y)
    y = Conv2D(kernel_size=kernel_size,
               strides=1,
               filters=filters,
               padding="same")(y)

    if downsample:
        x = Conv2D(kernel_size=1,
                   strides=2,
                   filters=filters,
                   padding="same")(x)
    out = Add()([x, y])
    out = relu_bn(out)
    return out

Output:
What is the idea behind using nn.Identity for residual learning?

There is none (almost, see the end of the post), all nn.Identity does is forwarding the input given to it (basically no-op).
As shown in PyTorch repo issue you linked in comment this idea was first rejected, later merged into PyTorch, due to other use (see the rationale in this PR). This rationale is not connected to ResNet block itself, see end of the answer.
ResNet implementation
Easiest generic version I can think of with projection would be something along those lines:
class Residual(torch.nn.Module):
    def __init__(self, module: torch.nn.Module, projection: torch.nn.Module = None):
        super().__init__()
        self.module = module
        self.projection = projection

    def forward(self, inputs):
        output = self.module(inputs)
        if self.projection is not None:
            inputs = self.projection(inputs)
        return output + inputs

You can pass as module things like two stacked convolutions and add 1x1 convolution (with padding or with strides or something) as projection module.
For tabular data you could use this as module (assuming your input has 50 features):
torch.nn.Sequential(
    torch.nn.Linear(50, 50),
    torch.nn.ReLU(),
    torch.nn.Linear(50, 50),
    torch.nn.ReLU(),
    torch.nn.Linear(50, 50),
)

Basically, all you have to do is is add input to some module to it's output and that is it.
Rationale behing nn.Identity
It might be easier to construct neural networks (and read them afterwards), example for batch norm (taken from aforementioned PR):
batch_norm = nn.BatchNorm2d
if dont_use_batch_norm:
    batch_norm = Identity

Now you can use it with nn.Sequential easily:
nn.Sequential(
    ...
    batch_norm(N, momentum=0.05),
    ...
)

And when printing the network it always has the same number of submodules (with either BatchNorm or Identity) which also makes the whole thing a little smoother IMO.
Another use case, mentioned here might be removing parts of existing neural networks:
net = tv.models.alexnet(pretrained=True)
# Assume net has two parts
# features and classifier
net.classifier = Identity()

Now, instead of running net.features(input) you can run net(input) which might be easier for others to read as well.


Instruction: PyTorch VERY different results on different machines using docker and CPU

Input:
I am trying to create some CI tests for a deep learning model and therefore created a Dockerfile to install PyTorch and all my requirements and finally run the tests.
FROM pytorch/pytorch
ADD . / project/
RUN (cd project/; pip install -r requirements.txt)
CMD ( cd project/; pytest -v --cov=my_project)

The tests are basically computing an image from 0 - 1  and comparing it to a reference image (saved as npy). The test is checking
if the avg L2 norm of the pixels is below a threshold of 1e-7.
diff_image = np.linalg.norm(target_image_np - reference_image_np, axis=2)
avg_error = np.mean(diff_image)
assert avg_error < 1e-7

The tests pass 12/15 test cases. However, 3 cases are failing quite badly.
=========================== short test summary info ============================
FAILED test_nst.py::test_nst_gatys - assert 0.0021541715 < 1e-07
FAILED test_nst.py::test_nst_gatys_style - assert 0.12900369 < 1e-07
FAILED test_nst.py::test_nst_wct - assert 0.027357593 < 1e-07
=================== 3 failed, 12 passed in 670.27s (0:11:10) ===================

The weird thing is that this ONLY happens on the CI server. On my local machine, all tests pass. Does anybody have an idea why this is happening? As far as I know, using the CPU as well as fixed seeds should return at least results which differ only numerically.
Thanks for any feedback!

Output:
I found a solution myself. It is required to set the number of threads to one. Finally, this is all the code required to get reproducible results on different machines.
    np.random.seed(42)
    torch.manual_seed(42)
    os.environ["PYTHONHASHSEED"] = "42"
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.set_num_threads(1)


Instruction: Top K indices of a multi-dimensional tensor

Input:
I have a 2D tensor and I want to get the indices of the top k values. I know about pytorch's topk function. The problem with pytorch's topk function is, it computes the topk values over some dimension. I want to get topk values over both dimensions.
For example for the following tensor
a = torch.tensor([[4, 9, 7, 4, 0],
        [8, 1, 3, 1, 0],
        [9, 8, 4, 4, 8],
        [0, 9, 4, 7, 8],
        [8, 8, 0, 1, 4]])

pytorch's topk function will give me the following.
values, indices = torch.topk(a, 3)

print(indices)
# tensor([[1, 2, 0],
#        [0, 2, 1],
#        [0, 1, 4],
#        [1, 4, 3],
#        [1, 0, 4]])

But I want to get the following
tensor([[0, 1],
        [2, 0],
        [3, 1]])

This is the indices of 9 in the 2D tensor.
Is there any approach to achieve this using pytorch?

Output:
v, i = torch.topk(a.flatten(), 3)
print (np.array(np.unravel_index(i.numpy(), a.shape)).T)

Output:
[[3 1]
 [2 0]
 [0 1]]

Flatten and find top k
Convert 1D indices to 2D using unravel_index


Instruction: Loading images in PyTorch

Input:
I am new to PyTorch and working on a GAN model. I want to load my image dataset. The way its done using Keras is:
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import load_img

def load_images(path, size=(128,128)):
data_list = list()
# enumerate filenames in directory, assume all are images
for filename in listdir(path):
    # load and resize the image
    pixels = load_img(path + filename, target_size=size)
    # convert to numpy array
    pixels = img_to_array(pixels)
    # store.
    data_list.append(pixels)
return asarray(data_list)
# dataset path
path = 'mypath/'
# load dataset A
dataA = load_images(path + 'A/')
dataAB = load_images(path + 'B/')

I want to know how to do the same in PyTorch.
Any help is appreciated. Thanks

Output:
import torchvision, torch
from torchvision import datasets, models, transforms

def load_training(root_path, dir, batch_size, kwargs):

    transform = transforms.Compose(
        [transforms.Resize([256, 256]),
         transforms.RandomCrop(224),
         transforms.RandomHorizontalFlip(),
         transforms.ToTensor()])
    data = datasets.ImageFolder(root=root_path + dir, transform=transform)
    train_loader = torch.utils.data.DataLoader(data, batch_size=batch_size, shuffle=True, drop_last=True, **kwargs)

    return train_loader

I hope it'll work ...


Instruction: what is the function of default_loader in torch?

Input:
    import os
    import pandas as pd
    import numpy as np
    from torchvision.datasets.folder import default_loader
    from torchvision.datasets.utils import download_url
    from torch.utils.data import Dataset

 class Sample_Class(Dataset):
      def __init__(self,root,train=True,transform=None,loader=default_loader):
        self.root = os.path.expanduser(root)
        self.transform = transform
        self.loader = default_loader

In the above code snippet , what is the significance of loader=default_loader, what exactly does that do?

Output:
This Sample_Class is likely imitating the behavior of ImageFolder, DatasetFolder, and ImageNet. The function should take a filename as input and return either a PIL.Image or accimage.Image depending on the selected image backend.
The default_loader function is defined in torchvision/datasets/folder.py
def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGB')

def accimage_loader(path):
    import accimage
    try:
        return accimage.Image(path)
    except IOError:
        # Potentially a decoding problem, fall back to PIL.Image
        return pil_loader(path)

def default_loader(path):
    from torchvision import get_image_backend
    if get_image_backend() == 'accimage':
        return accimage_loader(path)
    else:
        return pil_loader(path)

Note : default_loader by default will be PIL reader


Instruction: Sliding Window on 2D Tensor using PyTorch

Input:
How can we use a sliding window on a 2D PyTorch tensor t with shape (6, 10) such that we end up with a 3D PyTorch tensor with shape (3, 4, 10)?
For example, if we have the tensor t:
t = torch.range(1, 6*10).reshape((7, 10))

tensor([[ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.],
        [11., 12., 13., 14., 15., 16., 17., 18., 19., 20.],
        [21., 22., 23., 24., 25., 26., 27., 28., 29., 30.],
        [31., 32., 33., 34., 35., 36., 37., 38., 39., 40.],
        [41., 42., 43., 44., 45., 46., 47., 48., 49., 50.],
        [51., 52., 53., 54., 55., 56., 57., 58., 59., 60.]]

how can we reshape it (using PyTorch) such that we get the tensor:
tensor([[ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.],
        [11., 12., 13., 14., 15., 16., 17., 18., 19., 20.],
        [21., 22., 23., 24., 25., 26., 27., 28., 29., 30.],
        [31., 32., 33., 34., 35., 36., 37., 38., 39., 40.]],

        [[11., 12., 13., 14., 15., 16., 17., 18., 19., 20.],
        [21., 22., 23., 24., 25., 26., 27., 28., 29., 30.],
        [31., 32., 33., 34., 35., 36., 37., 38., 39., 40.],
        [41., 42., 43., 44., 45., 46., 47., 48., 49., 50.]],

        [[21., 22., 23., 24., 25., 26., 27., 28., 29., 30.],
        [31., 32., 33., 34., 35., 36., 37., 38., 39., 40.],
        [41., 42., 43., 44., 45., 46., 47., 48., 49., 50.],
        [51., 52., 53., 54., 55., 56., 57., 58., 59., 60.]])

Output:
Looks like a case for unfold:
t.unfold(0,4,1).transpose(2,1)

Output:
tensor([[[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
         [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
         [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
         [30, 31, 32, 33, 34, 35, 36, 37, 38, 39]],

        [[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
         [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
         [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
         [40, 41, 42, 43, 44, 45, 46, 47, 48, 49]],

        [[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
         [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
         [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
         [50, 51, 52, 53, 54, 55, 56, 57, 58, 59]]])


Instruction: Found no NVIDIA driver on your system error on WSL2 conda environment with Python 3.7

Input:
I have a Nvidia 1080Ti GPU, and I want to run Pytorch on WSL2, but I got error "Found no NVIDIA driver on your system" but I did installed the NVIDIA driver. Here is the step I did.

I installed WSL2, and installed NVIDIA driver for Cuda on WSL from GeForce Driver:
https://developer.nvidia.com/cuda/wsl/download

I activate a clean conda environment with Python 3.7

Then I run the Pytorch installation:
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

Then the error occurred saying Found no NVIDIA driver. I came across a post on Pytorch forum, and someone did get it to run in a similar settings: Ubuntu 18.04 + Conda + Pytorch
https://discuss.pytorch.org/t/found-no-nvidia-driver-on-your-system-but-its-there/35063/4
I don't have multiple GPUs, so I don't know how to get my driver recognized in WSL2. Thank you for any thoughts!

Output:
What's your Windows version? (Run winver.exe)
You need to run a Windows Insider build 20145 or superior in order to use CUDA in WSL2.
You will know the gpu is detected if /dev/dxg file exists.


Instruction: Libtorch C++ - no matching member function for call to 'size' for InterpolateFuncOptions

Input:
Using Libtorch 1.6.0 in C++, I get the following error:
error: no matching member function for call to 'size'

My line is the following:
image = F::interpolate(image, F::InterpolateFuncOptions().size({target_height, target_width}).mode(torch::kNearest));

But in the documentation it seems correct... Any idea?
Thanks in advance

Output:
You should wrap it with std::vector like this:
image = F::interpolate(image,
        F::InterpolateFuncOptions()
        .size(std::vector<>{target_height, target_width})
        .mode(torch::kNearest));

Reason for this is size has no overloaded call for std::initializer_list that you were trying to use (see size docs here).


Instruction: Using Captum with Pytorch Lightning?

Input:
So I tried to use Captum with PyTorch Lightning. I am having issues when passing the Module to Captum, since it seems to do weird reshaping of the tensors.
For example in the below minimal example, the lightning code works easy and well.
But when I use IntegratedGradient with "n_step>=1" I get an issue.
The code of the LighningModule is not that important I would say, I wonder more at the code line at the very bottom.
Does anyone know how to work around this?
from captum.attr import IntegratedGradients
from torch import nn, optim, rand, sum as tsum, reshape, device
import torch.nn.functional as F
from pytorch_lightning import seed_everything, LightningModule, Trainer
from torch.utils.data import DataLoader, Dataset

SAMPLE_DIM = 3

class CustomDataset(Dataset):
    def __init__(self, samples=42):
        self.dataset = rand(samples, SAMPLE_DIM).cuda().float() * 2 - 1

    def __getitem__(self, index):
        return (self.dataset[index], (tsum(self.dataset[index]) > 0).cuda().float())

    def __len__(self):
        return self.dataset.size()[0]

class OurModel(LightningModule):
    def __init__(self):
        super(OurModel, self).__init__()
        # Network layers
        self.linear = nn.Linear(SAMPLE_DIM, 2048)
        self.linear2 = nn.Linear(2048, 1)
        self.output = nn.Sigmoid()
        # Hyper-parameters, that we will auto-tune using lightning!
        self.lr = 0.001
        self.batch_size = 512

    def forward(self, x):
        x = self.linear(x)
        x = self.linear2(x)
        output = self.output(x)
        return reshape(output, (-1,))

    def configure_optimizers(self):
        return optim.Adam(self.parameters(), lr=self.lr)

    def train_dataloader(self):
        loader = DataLoader(CustomDataset(samples=1000), batch_size=self.batch_size, shuffle=True)
        return loader

    def training_step(self, batch, batch_nb):
        x, y = batch
        loss = F.binary_cross_entropy(self(x), y)
        return {'loss': loss, 'log': {'train_loss': loss}}

if __name__ == '__main__':
    seed_everything(42)
    device = device("cuda")
    model = OurModel().to(device)
    trainer = Trainer(max_epochs=2, min_epochs=1, auto_lr_find=False,
                      progress_bar_refresh_rate=10)
    trainer.fit(model)
    # ok Now the Problem
    test_input = CustomDataset(samples=1).__getitem__(0)[0].requires_grad_()
    ig = IntegratedGradients(model)
    attr, delta = ig.attribute(test_input, target=1, return_convergence_delta=True)

Output:
The solution was to wrap the forward function. Make sure that the shape going into the mode.foward() is correct!
# Solution is this wrapper function
def modified_f(in_vec):
    # Shape here is wrong
    print("IN:", in_vec.size())
    x = torch.reshape(in_vec, (int(in_vec.size()[0]/SAMPLE_DIM), SAMPLE_DIM))
    print("x:", x.size())

    res = model.forward(x)
    print("res:", res.size())
    res = torch.reshape(res, (res.size()[0], 1))
    print("res2:", res.size())

    return res

ig = IntegratedGradients(modified_f)
attr, delta = ig.attribute(test_input, return_convergence_delta=True, n_steps=STEP_AMOUNT)


Instruction: pytorch: compute vector-Jacobian product for vector function

Input:
Good day!
I am trying to grasp torch.autograd basics. In particular I want to test this statement from https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py

So my idea is to construct a vector function, say:
(y_1; y_2; y_3) = (x_1*x_1 + x_2; x_2 * x_2 + x_3; x_3 * x_3)
Then count Jacobian matrix at point (1,1,1) and multiply it on vector (3, 5, 7).
Jacobian = (2x_1; 1.   ; 0.  )
(0.  ; 2x_2 ; 1.  )
(0.  ; 0.   ; 2x_3)
I am expecting result Jacobian(x=(1,1,1)) * v = (6+5, 10 + 7, 2 * 7) = (11, 17, 14).
Now below is my attempt to do it in pytorch:
import torch

x = torch.ones(3, requires_grad=True)
print(x)

y = torch.tensor([x[0]**2 + x [1], x[1]**2 + x[2], x[2]**2], requires_grad=True)
print(y)

v = torch.tensor([3, 5, 7])

y.backward(v)
x.grad

which gives not expected result (2., 2., 1.). I think I define tensor y in a wrong way. If I simply do y = x * 2, then gradient will work, but what about creating more complex tensor like in this case?
Thank you.

Output:
You should not define tensor y by torch.tensor(), torch.tensor() is a tensor constructor, not an operator, so it is not trackable in the operation graph. You should use torch.stack() instead.
Just change that line to:
y = torch.stack((x[0]**2+x[1], x[1]**2+x[2], x[2]**2))

the result of x.grad should be tensor([ 6., 13., 19.])


Instruction: How to restructure the output tensor of a cnn layer for use by a linear layer in a simple pytorch model

Input:
Given a pytorch input dataset  with dimensions:
dat.shape = torch.Size([128, 3, 64, 64])

This is a supervised learning problem: we have a separate labels.txt file containing one of C classes for each input observation.  The value of C is calculated by the number of distinct values in the labeles file and is presently in the single digits.
I could use assistance on how to mesh the layers of a simple mix of convolutional and linear layers network that is performing multiclass classification.  The intent is to pass through:

two cnn layers with maxpooling after each
a linear "readout" layer
softmax activation before the output/labels

Here is the core of my (faulty/broken) network.  I am unable to determine the proper size/shape required of:
 Output of Convolutional layer -> Input of Linear [Readout] layer

class CNNClassifier(torch.nn.Module):

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 16, 3)
        self.maxpool = nn.MaxPool2d(kernel_size=3,padding=1)
        self.conv2 = nn.Conv2d(16, 32, 3)
        self.linear1 = nn.Linear(32*16*16, C)
        self.softmax1 = nn.LogSoftmax(dim=1)

    def forward(self, x):
        x = self.conv1(x)
        x = self.maxpool(F.leaky_relu(x))
        x = self.conv2(x)
        x = self.maxpool(F.leaky_relu(x))
        x = self.linear1(x)  # Size mismatch error HERE
        x = self.softmax1(x)
        return x

Training of the model is started by :
        Xout = model(dat)

This results in :

RuntimeError: size mismatch, m1: [128 x 1568], m2: [8192 x 6]

at the linear1 input.  What is needed here ? Note I have seen uses of wildcard input sizes e.g via a view:
    ..
    x = x.view(x.size(0), -1)
    x = self.linear1(x)  # Size mismatch error HERE

If that is included then the error changes to

RuntimeError: size mismatch, m1: [28672 x 7], m2: [8192 x 6]

Some pointers on how to think about and calculate the cnn layer / linear layer input/output sizes would be much appreciated.

Output:
The error
You have miscalculated the output size from convolutional stack. It is actually [batch, 32, 7, 7] instead of [batch, 32, 16, 16].
You have to use reshape (or view) as output from Conv2d has 4 dimensions ([batch, channels, width, height]), while input to nn.Linear is required to have 2 dimensions ([batch, features]).
Use this for nn.Linear:
self.linear1 = nn.Linear(32 * 7 * 7, C)

And this in forward:
x = self.linear1(x.view(x.shape[0], -1))

Other possibilities
Current new architectures use pooling across channels (usually called global pooling). In PyTorch there is an torch.nn.AdaptiveAvgPool2d (or Max pooling). Using this approach allows you to have variable size of height and width of your input image as only one value per channel is used as input to nn.Linear. This is how it looks:
class CNNClassifier(torch.nn.Module):
    def __init__(self, C=10):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 16, 3)
        self.maxpool = nn.MaxPool2d(kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, 3)
        self.pooling = torch.nn.AdaptiveAvgPool2d(output_size=1)
        self.linear1 = nn.Linear(32, C)
        self.softmax1 = nn.LogSoftmax(dim=1)

    def forward(self, x):
        x = self.conv1(x)
        x = self.maxpool(F.leaky_relu(x))
        x = self.conv2(x)
        x = self.maxpool(F.leaky_relu(x))
        x = self.linear1(self.pooling(x).view(x.shape[0], -1))
        x = self.softmax1(x)
        return x

So now images of torch.Size([128, 3, 64, 64]) and torch.Size([128, 3, 128, 128]) can be passed to the network.