Instruction: How to implement time-distributed dense (TDD) layer in PyTorch

Input:
In some deep learning models which analyse temporal data (e.g. audio, or video), we use a "time-distributed dense" (TDD) layer. What this creates is a fully-connected (dense) layer which is applied separately to every time-step.

In Keras this can be done using the TimeDistributed wrapper, which is actually slightly more general. In PyTorch it's been an open feature request for a couple of years.

How can we implement time-distributed dense manually in PyTorch?

Output:
Specifically for time-distributed dense (and not time-distributed anything else), we can hack it by using a convolutional layer.

Look at the diagram you've shown of the TDD layer. We can re-imagine it as a convolutional layer, where the convolutional kernel has a "width" (in time) of exactly 1, and a "height" that matches the full height of the tensor. If we do this, while also making sure that our kernel is not allowed to move beyond the edge of the tensor, it should work:

self.tdd = nn.Conv2d(1, num_of_output_channels, (num_of_input_channels, 1))

You may need to do some rearrangement of tensor axes. The "input channels" for this line of code are in fact coming from the "freq" axis (the "image's y axis") of your tensor, and the "output channels" will indeed be arranged on the "channel" axis. (The "y axis" of the output will be a singleton dimension of height 1.)


Instruction: With PyTorch, how is my Conv1d dimension reducing when I have padding?

Input:
My conv module is:

        return torch.nn.Sequential(
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=1,
                padding=1
            ),
            torch.nn.ReLU(),
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=2,
                padding=1
            ),
            torch.nn.ReLU(),
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=4,
                padding=1
            ),
            torch.nn.ReLU()
        )

And in forward, I have:

down_out = self.downscale_time_conv(inputs)

inputs has a .size of torch.Size([8, 161, 24]). I'd expect down_out to have the same size, but instead it has: torch.Size([8, 161, 23])

Where did that last element go?

Output:
The answer can be found on Pytorch documentation online (here). For every operation the output shape is expressed with respect to the input parameters:

For each conv1D:

- L1 = 25 → int((24 + 2*1 - 1*(2 - 1) - 1) / 1 + 1)
- L2 = 25 → int((25 + 2*1 - 2*(2 - 1) - 1) / 1 + 1)
- L3 = 23 → int((25 + 2*1 - 4*(2 - 1) - 1) / 1 + 1)

Do not forget that Lin is the previous size.


Instruction: Can I program a GPU when it is the system's only graphics card?

Input: Can I run PyTorch or Tensorflow on Windows on a GPU that is also acting as the system's graphics card (e.g. there is no graphics built-in to a Ryzen 3600 CPU)? If so, is there any downside, or would I be better off getting a CPU with built-in graphics?

Output:
Yes, it is possible to run i.e. Tensorflow on GPU while also using the GPU for your system. You do not need a 2nd graphics-card or integrated GPU.
Keep in mind, your graphicscard will share memory and processing-power between all your programs. GPU-intensive work might slow down the fps on your system and the other way around. Also keep an eye on the memory usage.

I had tensorflow_gpu with a multi-layer CNN running, while playing AAA-Games (i.e. GTA V) using a Ryzen 3600. It worked on my super old NVIDIA GTX 260 (2GB memory) but it crashed quite often cause of the limited memory. Upgraded to a GTX 1080 with 8GB and it worked quite well. Needless to say, you can always fill up your GPU-memory and crash, no matter the size.


Instruction: Loading image data from pandas to pytorch

Input:
I am completely new to pytorch and have previously worked on keras and fastai.
Currently trying an image regression task and the challenge is I have to load the data from pandas dataframe.
Data frame structure:

ID   Path   Score
fig1  /folder/fig1.jpg  2
fig2  /folder/fig2.jpg  3
.....

I have previously worked on loading images to pytorch directly from folders because it was a simple classification task but kind of stuck now.

I looked into pytorch forums but didn't quiet understand how to implement.
Any help would be appreciated.

Output:
Datasets

You have to use torch.utils.data.Dataset structure to define it.
Here is how you can do it in plain pytorch (I'm using pillow to load the images and torchvision to transform them to torch.Tensor objects):

import torch
import torchvision
from PIL import Image

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, dataframe):
        self.dataframe = dataframe

    def __len__(self):
        return len(self.dataframe)

    def __getitem__(self, index):
        row = self.dataframe.iloc[index]
        return (
            torchvision.transforms.functional.to_tensor(Image.open(row["Path"])),
            row["Score"],
        )

dataset = MyDataset(dataframe)

Alternatively, you can use torchdata (disclaimer: shameless self-promotion as I'm the author...) which allows you to decouple Path and Scores like this:

import torchvision
from PIL import Image

import torchdata

class ImageDataset(torchdata.datasets.FilesDataset):
    def __getitem__(self, index):
        return Image.open(self.files[index])

class Labels(torchdata.Dataset):
    def __init__(self, scores):
        super().__init__()
        self.scores = scores

    def __len__(self):
        return len(self.scores)

    def __getitem__(self, index):
        return self.scores[index]

# to_numpy for convenience
# I assume all your images are in /folder and have *.jpg extension
dataset = ImageDataset.from_folder("/folder", regex="*.jpg").map(
    torchvision.transforms.ToTensor()
) | Labels(dataframe["Score"].to_numpy())

(or you could implement it just like in regular pytorch but inheriting from torchdata.Dataset and calling super().__init__() in the constructor).

torchdata allows you to cache your images easily or apply some other transformations via .map as shown there, check github repository for more info or ask in the comment.

DataLoader

Either way you choose you should wrap your dataset in torch.utils.data.DataLoader to create batches and iterate over them, like this:

dataloader = torch.utils.data.DataLoader(dataset, batch_size=64, shuffle=True)

for images, scores in dataloader:
    # Rest of your code to train neural network or smth
    ...

Do with those images and scores what you want in the loop.


Instruction: PyTorch: Speed up data loading

Input:
I am using densenet121 to do cat/dog detection from Kaggle dataset. I enabled cuda and it appears that training is very fast. However, the data loading (or perhaps processing) appears to be very slow. Are there some ways to speed it up? I tried to play witch batch size, that didn't provide much help. I also changed num_workers from 0 to some positive numbers. Going from 0 to 2 reduces loading time by perhaps 1/3, increasing by more doesn't have additional effect. Are there some other ways I can speed loading things up?

This is my rough code (I am focused on learning, so it's not very organized):

import matplotlib.pyplot as plt

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms, models

data_dir = 'Cat_Dog_data'

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.5, 0.5, 0.5],
                                                            [0.5, 0.5, 0.5])])
test_transforms = transforms.Compose([transforms.Resize(255),
                                      transforms.CenterCrop(224),
                                      transforms.ToTensor()])

# Pass transforms in here, then run the next cell to see how the transforms look
train_data = datasets.ImageFolder(data_dir + '/train',
                                  transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)

trainloader = torch.utils.data.DataLoader(train_data, batch_size=64,
                                          num_workers=16, shuffle=True,
                                          pin_memory=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=64,
                                         num_workers=16)

model = models.densenet121(pretrained=True)

# Freeze parameters so we don't backprop through them
for param in model.parameters():
    param.requires_grad = False

from collections import OrderedDict

classifier = nn.Sequential(OrderedDict([
    ('fc1', nn.Linear(1024, 500)),
    ('relu', nn.ReLU()),
    ('fc2', nn.Linear(500, 2)),
    ('output', nn.LogSoftmax(dim=1))
]))

model.classifier = classifier
model.cuda()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

epochs = 30
steps = 0

import time

device = torch.device('cuda:0')

train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    count = 0
    total_start = time.time()
    for images, labels in trainloader:
        start = time.time()
        images = images.cuda()
        labels = labels.cuda()

        optimizer.zero_grad()

        log_ps = model(images)
        loss = criterion(log_ps, labels)
        loss.backward()
        optimizer.step()
        elapsed = time.time() - start

        if count % 20 == 0:
            print("Optimized elapsed: ", elapsed, "count:", count)
            print("Total elapsed ", time.time() - total_start)
            total_start = time.time()
        count += 1

        running_loss += loss.item()
    else:
        test_loss = 0
        accuracy = 0
        for images, labels in testloader:
            images = images.cuda()
            labels = labels.cuda()
            with torch.no_grad():
                model.eval()
                log_ps = model(images)
                test_loss += criterion(log_ps, labels)
                ps = torch.exp(log_ps)
                top_p, top_class = ps.topk(1, dim=1)
                compare = top_class == labels.view(*top_class.shape)
                accuracy += compare.type(torch.FloatTensor).mean()
        model.train()
        train_losses.append(running_loss / len(trainloader))
        test_losses.append(test_loss / len(testloader))

        print("Epoch: {}/{}.. ".format(e + 1, epochs),
              "Training Loss: {:.3f}.. ".format(
                  running_loss / len(trainloader)),
              "Test Loss: {:.3f}.. ".format(test_loss / len(testloader)),
              "Test Accuracy: {:.3f}".format(accuracy / len(testloader)))

Output:
torchvision 0.8.0 version or greater
Actually torchvision now supports batches and GPU when it comes to transformations (this is done on torch.Tensors instead of PIL images), so one should use it as an initial improvement.
See here for more info about this release. Also those act as torch.nn.Module, hence can be used inside a model, for example:
transforms = torch.nn.Sequential(
    T.RandomCrop(224),
    T.RandomHorizontalFlip(p=0.3),
    T.ConvertImageDtype(torch.float),
    T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
)

Furthermore, those operations could be JITed possibly improving the performance even further.
torchvision < 0.8.0 (original answer)
Increasing batch_size won't help as torchvision performs transform on single image while it's loaded from your disk.
There are a couple of ways one could speed up data loading with increasing level of difficulty:

Improve image loading times
Load & normalize images and cache in RAM (or on disk)
Produce transformations and save them to disk
Apply non-cache'able transforms (rotations, flips, crops) in batched manner
Prefetching

1. Improve image loading
Easy improvements can be gained by installing Pillow-SIMD instead of original pillow. It is a drop-in replacement and could be faster (or so is claimed at least for Resize which you are using).
Alternatively, you could create your own data loading and processing with OpenCV as some say it's faster or check albumentations (though can't tell you whether those will improve the performance and might be a lot of time wasted for no gain except learning experience).
2. Load & normalize images & cache
You can use Python's LRU Cache functionality to cache some outputs.
You can also use torchdata which acts almost exactly like PyTorch's torch.utils.data.Dataset but allows caching to disk or in RAM (or mixed modes) with simple cache() on torchdata.Dataset (see github repository, disclaimer: i'm the author).
Remember: you have to load and normalize images, cache and after that use RandomRotation, RandomResizedCrop and RandomHorizontalFlip (as those change each time they are run).
3. Produce transformations and save them to disk
You would have to perform a lot of transformations on images, save them to disk and use this enhanced dataset afterwards. Once again that could be done with torchdata but it's really wasteful when it comes to I/O and hard drive and very inelegant solution. Furthermore it's "static" so the data would only last your for X epochs, it wouldn't be "infinite" generator with augmentations.
4. Batched transformations
torchvision does not support it so you would have to write those functions on your own. See this issue for justification. AFAIK no other 3rd party provides it either. For large batches it should speed up things but implementation is open question I think (correct me if I'm wrong).
5. Prefetch
IMO would be hardest to implement (though a really good idea for the project come to think about it). Basically you load data for the next iteration when your model trains. torch.utils.data.DataLoader does provide it, though there are some concerns (like workers pausing after their data got loaded). You can read PyTorch thread about it (not sure about it as I didn't verify on my own). Also, a lot of valuable insight provided by this comment and this blog post (though not sure how up to date those are).
All in all to substantially improve data loading you would need to get your hands quite dirty (or maybe there are libraries doing this some of those for PyTorch, if so,I would love to know about them).
Also remember to profile your changes, see torch.nn.bottleneck
EDIT: DALI project might be worth checking out, though AFAIK it has some problems with RAM memory growing linearly with number of epochs.


Instruction: Pytorch: RGB value ranges 0-1 after rescaling, How do I normalize images?

Input:
I wrote a class to rescale images, but the RGB value became ranging from 0 to 1 after preocessing. What happened to the RGB which intuitively should be ranging from 0-255 ? Following are the Rescale class and the RGB values after rescaling.

Question:

Do I still need a Min-Max Normalization, map the RGB value to 0-1?

How do I apply transforms.Normalization, where do I put the Normalization, before or after the Rescale, how do I calculate the mean and variance, use the RGB value ranging from 0-255 or 0-1？

Thanks for your time!

class Rescale(object):
    def __init__(self, output_size):
        assert isinstance(output_size, (int, tuple))
        self.output_size = output_size

    def __call__(self, sample):
        image, anno = sample['image'], sample['anno']

        # get orginal width and height of image
        h, w = image.shape[0:2]
        # if output_size is an integer

        if isinstance(self.output_size, int):
            if h > w:
                new_h, new_w = h * self.output_size / w, self.output_size
            else:
                new_h, new_w = self.output_size / h, w * self.output_size / h

        # if output size is a tuple (a, b)
        else:
            new_h, new_w = self.output_size
        new_h, new_w = int(new_h), int(new_w)

        image = transform.resize(image, (new_h, new_w))
        return {'image': image, 'anno': anno}

[[[0.67264216 0.50980392 0.34503034]
  [0.67243905 0.51208121 0.34528431]
  [0.66719145 0.51817184 0.3459951 ]
  ...
  [0.23645098 0.2654311  0.3759458 ]
  [0.24476471 0.28003857 0.38963938]
  [0.24885877 0.28807445 0.40935877]]

 [[0.67465196 0.50994608 0.3452402 ]
  [0.68067157 0.52031373 0.3531848 ]
  [0.67603922 0.52732436 0.35839216]
  ...
  [0.23458333 0.25195098 0.36822142]
  [0.2461343  0.26886127 0.38314558]
  [0.2454384  0.27233056 0.39977664]]

 [[0.67707843 0.51237255 0.34766667]
  [0.68235294 0.5219951  0.35553024]
  [0.67772059 0.52747687 0.35659176]
  ...
  [0.24485294 0.24514568 0.36592999]
  [0.25407436 0.26205475 0.38063318]
  [0.2597007  0.27202914 0.40214216]]

 ...

[[[172 130  88]
  [172 130  88]
  [172 130  88]
  ...
  [ 63  74 102]
  [ 65  76 106]
  [ 67  77 112]]

 [[173 131  89]
  [173 131  89]
  [173 131  89]
  ...
  [ 65  74 103]
  [ 64  75 105]
  [ 63  73 108]]

 [[173 131  89]
  [174 132  90]
  [174 132  90]
  ...
  [ 63  72 101]
  [ 62  71 102]
  [ 61  69 105]]
  ...

Output:
You can use torchvision to accomplish this.

transform = transforms.Compose([
    transforms.Resize(output_size),
    transforms.ToTensor(),
])

This requires a PIL image as input. It will return the tensor in [0, 1] range.You may also add mean-standard normalization as below

transform = transforms.Compose([
    transforms.Resize(output_size),
    transforms.ToTensor(),
    transforms.Normalize(mean, std),
])

Here mean and std are per channel mean and standard deviation of all pixels of all images in the training set. You need to calculate them after resizing all images and converting to torch Tensor. One way to do this would be to apply first two transformation (resize and ToTensor) and then calculate mean and std over all training images like this

x = torch.concatenate([train_data[i] for i in range(len(train_data))])
mean = torch.mean(x, dim=(0, 1))
std = torch.std(x, dim=(0, 1))

Then you use this mean and std value with Normalize transorm above.


Instruction: How to apply conditions for rows in a tensor where there is boolean values

Input:
I have the following tensor:

predictions = torch.tensor([[ True, False, False],
                            [False, False,  True],
                            [False,  True,  True],
                            [ True, False, False]])

I applied conditions along the axis like below.

new_pred= []

if predictions == ([True,False,False]):
       new_pred = torch.Tensor(0)
if predictions == ([False,False,True]):
       new_pred = torch.Tensor(2)
if predictions == ([False,True,True]):
       new_pred = torch.Tensor(2)

So I want the final output (new_pred) to be:
tensor([0, 2, 2, 0])

But I am getting a blank [] for the new_pred tensor. I think my logic must be flawed since nothing is getting stored in the new_pred. Can someone help me write this logic accurately?

Output:
The type of predictions is torch.Tensor while ([True, False, False]) is a list, first, you have to make sure both sides have the same type.

predictions == torch.tensor([True,False,False])
>>> tensor([[ True, True, True],
            [False, True, False],
            [False, False, False],
            [True, True, True]])

Then, you are still comparing a 2d tensor to a 1d tensor, which is ambiguous in an if statement, an easy way to fix this would be to write a for loop, compare each row of the predictions to the conditions and append the result to the new_pred list. Note that you will be comparing two booleans tensors with the size of three, therefore, you have to make sure the result of the comparison is True for all of the cells.

predictions = torch.tensor([[ True, False, False],
                            [False, False,  True],
                            [False,  True,  True],
                            [ True, False, False]])

conditions = torch.tensor([[True,False,False],
                            [False,False,True],
                            [False,True,True]])
new_predict = []
for index in range(predictions.size(0)):
    if (predictions[index] == conditions[0]).all():
        new_predict.append(0)
    # ...

Alternatively, you can use slicing to achieve your expected result without any for loop.


Instruction: Unable to update PyTorch 1.4.0 to 1.5.0 using Conda

Input:
When I tried to udpdate PyTorch from 1.4.0 to 1.5.0, Anaconda says that all the packages are already installed.

$ conda install -c pytorch pytorch torchvision

Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

$ conda list | grep -i torch

_pytorch_select           0.2                       gpu_0
pytorch                   1.4.0           py3.7_cuda10.0.130_cudnn7.6.3_0    pytorch
torchvision               0.5.0                py37_cu100    pytorch

I believe 1.5.0 is available in the pytorch channel

$ conda search -c pytorch pytorch=1.5.0

Loading channels: done
# Name                       Version           Build  Channel
pytorch                        1.5.0     py3.5_cpu_0  pytorch
pytorch                        1.5.0 py3.5_cuda10.1.243_cudnn7.6.3_0  pytorch
pytorch                        1.5.0 py3.5_cuda10.2.89_cudnn7.6.5_0  pytorch
pytorch                        1.5.0 py3.5_cuda9.2.148_cudnn7.6.3_0  pytorch
pytorch                        1.5.0     py3.6_cpu_0  pytorch
pytorch                        1.5.0 py3.6_cuda10.1.243_cudnn7.6.3_0  pytorch
pytorch                        1.5.0 py3.6_cuda10.2.89_cudnn7.6.5_0  pytorch
pytorch                        1.5.0 py3.6_cuda9.2.148_cudnn7.6.3_0  pytorch
pytorch                        1.5.0     py3.7_cpu_0  pytorch
pytorch                        1.5.0 py3.7_cuda10.1.243_cudnn7.6.3_0  pytorch
pytorch                        1.5.0 py3.7_cuda10.2.89_cudnn7.6.5_0  pytorch
pytorch                        1.5.0 py3.7_cuda9.2.148_cudnn7.6.3_0  pytorch
pytorch                        1.5.0     py3.8_cpu_0  pytorch
pytorch                        1.5.0 py3.8_cuda10.1.243_cudnn7.6.3_0  pytorch
pytorch                        1.5.0 py3.8_cuda10.2.89_cudnn7.6.5_0  pytorch
pytorch                        1.5.0 py3.8_cuda9.2.148_cudnn7.6.3_0  pytorch

Why is conda not updating PyTorch to 1.5.0?

Using Python 3.7.3 & conda 4.8.3 on Ubuntu 18.04

Thanks!

Output:
Install Validates Constraints

The Conda install first checks to see if a constraint is satisfied, rather than blindly trying to install the latest of everything. A better reading of the command:

conda install -c pytorch pytorch torchvision

would be

  With the pytorch channel prioritized, ensure that the currently activated environment has some version of pytorch and torchvision installed.

Your environment already satisfies this constraint, so there is nothing to do.

Updating Packages, or Constraints

If you want to update a package, then look into the conda update command or, if you know a minimum version you require, then specify it:

conda install -c pytorch pytorch[version='>=1.5'] torchvision

which effectively changes the constraint.

Better Practice (Recommended)

Best practice though is to simply make a new env when you require changes to packages. Every time one changes the packages in an env, one risks breaking/invalidating existing code.

conda create -n pytorch_1_5 -c pytorch pytorch torchvison

And this will grab the latest possible versions by default.


Instruction: question with fast.ai course lesson 8 g attribute

Input:
In the course fast.ai 2019 lesson 8, there is a weird g attribute used in back propagation, which i check for torch.Tensor this attribute doesn't exist. I tried to print the value of inp.g/out.g in call method but i got AttributeError: 'Tensor' object has no attribute 'g', but i am able to obtain the inp.g/out.g value before the assignment in backward, how does this g attribute works?

class Linear():
    def __init__(self, w, b):
        self.w, self.b = w, b

    def __call__(self, inp):
        print('in lin call')
        self.inp = inp
        self.out = inp@self.w + self.b
        try:
            print('out.g', self.out.g)
        except Exception as e:
            print('out.g dne yet')
        return self.out

    def backward(self):
        print('out.g', self.out.g)
        self.inp.g = self.out.g @ self.w.t()
        self.w.g = (self.inp.unsqueeze(-1) * self.out.g.unsqueeze(1)).sum(0)
        self.b.g = self.out.g.sum(0)

link to full code from the course

-update-

i am able to figure out the self.out.g value is exact same as the cost function MSE self.inp.g but still unable to figure out how the value is passed into the last linear layer.

class MSE():
    def __call__(self, inp, targ):
        self.inp = inp
        self.targ = targ
        self.out = (inp.squeeze() - targ).pow(2).mean()
        return self.out

    def backward(self):
        self.inp.g = 2. * (self.inp.squeeze() - self.targ).unsqueeze(-1) \
                        / self.targ.shape[0]
        print('in mse backward', self.inp.g)

class Model():
    def __init__(self, w1, b1, w2, b2):
        self.layers = [Lin(w1, b1), Relu(), Lin(w2, b2)]
        self.loss = Mse()

    def __call__(self, x, targ):
        for l in self.layers:
            x = l(x)
        return self.loss(x, targ)

    def backward(self):
        self.loss.backward()
        for l in reversed(self.layers):
            l.backward()

Output:
basically this has to deal with how python assignments works (pointers, similar to how C pointers works). after tracing the variables with id(variable name) i am able to figure out how things the g attribute comeby.

# ... in model (forward pass)...
    x = layer(x) # from linear layer >> return self.out and is assigned to x

# ...
    return self.loss(x, targ) # x is the same x (id) obtained from the model

# ========

# ... in model (backward pass) ...
    self.loss.backward() # this is how the self.inp.g came by

# ... in linear ...
    self.inp.g = self.out.g @ self.w.t()
    # this self.out.g is the same instance as self.inp.g from loss


Instruction: Understanding LSTM with a simple dataset

Input:
I wanted to make sure I understand LSTM so I implemented a dummy example using Pytorch framework.
As an input, I use sequences of consecutive numbers of length 10 and the value to predict is always the last number of sequence + 1. For instance:
x = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
y = 16

Since it's a very simple forecasting task, I expected the model to work well but I observe very poor performances. The model predicts a constant value by batch that keeps increasing during the training process.

I am wondering what I am missing. Below is the code I've made - any help would be highly appreciated.

from torch.utils.data import Dataset, TensorDataset, DataLoader, RandomSampler, SequentialSampler
import torch.nn as nn
import torch

class MyDataset(Dataset):

    def __init__(self):
        pass

    def __getitem__(self, index):
        x = torch.tensor([index-9,index-8,index-7,index-6,index-5,index-4,index-3,index-2,index-1,index])
        y = torch.tensor(index + 1)
        return x,y

    def __len__(self):
        return 1000

class LSTM(nn.Module):
    def __init__(self, hidden_layer_size=1, batch_size = 1):

        super().__init__()
        self.hidden_layer_size = hidden_layer_size
        self.batch_size = batch_size
        self.lstm = nn.LSTM(1, hidden_layer_size)
        self.linear = nn.Linear(10, 1)
        self.hidden_cell = (torch.zeros(1,self.batch_size,self.hidden_layer_size),
                            torch.zeros(1,self.batch_size,self.hidden_layer_size))

    def forward(self, input_seq):

        lstm_out, self.hidden_cell = self.lstm(input_seq.view(10 ,self.batch_size, -1), self.hidden_cell)
        predictions = self.linear(lstm_out.squeeze().T)
        return predictions

batch_size = 32
epochs = 1000

train = MyDataset()
sampler = RandomSampler(train)
train_dataloader = DataLoader(train, sampler=sampler, batch_size= batch_size , drop_last = True)

model = LSTM(batch_size = batch_size)
loss_function = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for e in range(epochs):
    for step, batch in enumerate(train_dataloader) :

        seq, labels = batch
        optimizer.zero_grad()

        model.hidden_cell = (torch.zeros(1, batch_size, model.hidden_layer_size),
                             torch.zeros(1, batch_size, model.hidden_layer_size))

        y_pred = model(seq.float())

        print(y_pred)

        single_loss = loss_function(y_pred, labels.float())
        single_loss.backward()
        optimizer.step()

Output:
There are multiple issues in your forward function. Take a look at the input that you are passing to the LSTM:

input_seq = input_seq.view(10 ,self.batch_size, -1)
print(input_seq[:, 0])

>>> tensor([[168.],
        [ 21.],
        [450.],
        [436.],
        [789.],
        [941.],
        [ -7.],
        [811.],
        [789.],
        [992.]])

This is a series of random numbers. You either have to transpose the input_seq or even better, pass batch_first=True to the LSTM constructor and just unsqueeze the input_seq before passing it to the LSTM.

You also have to update the lstm_out, the only operation that is needed now is to reshape it to [batch_size x (10 * hidden_size)].

Lastly, you need to squeeze the output of the linear layer.

Aside from those, the hidden size of the LSTM is too small, use 10 (or even 100) instead of one, only then the model converges in 1000 epochs.
Here is the updated code:

class LSTM(nn.Module):
    def __init__(self, hidden_layer_size=100, batch_size = 1):

        super().__init__()
        self.hidden_layer_size = hidden_layer_size
        self.batch_size = batch_size
        self.lstm = nn.LSTM(1, hidden_layer_size, batch_first=True)
        self.linear = nn.Linear(10 * hidden_layer_size, 1)
        self.hidden_cell = (torch.zeros(1,self.batch_size,self.hidden_layer_size),
                            torch.zeros(1,self.batch_size,self.hidden_layer_size))

    def forward(self, input_seq):
        batch_size = input_seq.size(0)
        input_seq = input_seq.unsqueeze(2)
        lstm_out, self.hidden_cell = self.lstm(input_seq, self.hidden_cell)
        lstm_out = lstm_out.reshape(batch_size, -1)
        predictions = self.linear(lstm_out).squeeze()
        return predictions


Instruction: SSD’s loss not decreasing in PyTorch

Input:
I am implementing SSD(Single shot detector) to study in PyTorch.
However, my custom training loss didn't decrease...
I've searched and tried various solution for week, but problem is still remaining.

What should I do?
My loss function is incorrect?

Here is my SSD300 model

SSD300(
  (feature_layers): ModuleDict(
    (conv1_1): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1_1): ReLU()
    (conv1_2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1_2): ReLU()
    (pool1): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
    (conv2_1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu2_1): ReLU()
    (conv2_2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu2_2): ReLU()
    (pool2): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
    (conv3_1): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3_1): ReLU()
    (conv3_2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3_2): ReLU()
    (conv3_3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3_3): ReLU()
    (pool3): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=True)
    (conv4_1): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu4_1): ReLU()
    (conv4_2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu4_2): ReLU()
    (conv4_3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu4_3): ReLU()
    (pool4): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
    (conv5_1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5_1): ReLU()
    (conv5_2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5_2): ReLU()
    (conv5_3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5_3): ReLU()
    (pool5): MaxPool2d(kernel_size=(3, 3), stride=(1, 1), padding=1, dilation=1, ceil_mode=False)
    (conv6): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(6, 6), dilation=(6, 6))
    (relu6): ReLU()
    (conv7): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1))
    (relu7): ReLU()
    (conv8_1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1))
    (relu8_1): ReLU()
    (conv8_2): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
    (relu8_2): ReLU()
    (conv9_1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1))
    (relu9_1): ReLU()
    (conv9_2): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
    (relu9_2): ReLU()
    (conv10_1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
    (relu10_1): ReLU()
    (conv10_2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1))
    (relu10_2): ReLU()
    (conv11_1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
    (relu11_1): ReLU()
    (conv11_2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1))
    (relu11_2): ReLU()
  )
  (localization_layers): ModuleDict(
    (loc1): Sequential(
      (l2norm_loc1): L2Normalization()
      (conv_loc1): Conv2d(512, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc1): ReLU()
    )
    (loc2): Sequential(
      (conv_loc2): Conv2d(1024, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc2): ReLU()
    )
    (loc3): Sequential(
      (conv_loc3): Conv2d(512, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc3): ReLU()
    )
    (loc4): Sequential(
      (conv_loc4): Conv2d(256, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc4): ReLU()
    )
    (loc5): Sequential(
      (conv_loc5): Conv2d(256, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc5): ReLU()
    )
    (loc6): Sequential(
      (conv_loc6): Conv2d(256, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc6): ReLU()
    )
  )
  (confidence_layers): ModuleDict(
    (conf1): Sequential(
      (l2norm_conf1): L2Normalization()
      (conv_conf1): Conv2d(512, 84, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf1): ReLU()
    )
    (conf2): Sequential(
      (conv_conf2): Conv2d(1024, 126, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf2): ReLU()
    )
    (conf3): Sequential(
      (conv_conf3): Conv2d(512, 126, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf3): ReLU()
    )
    (conf4): Sequential(
      (conv_conf4): Conv2d(256, 126, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf4): ReLU()
    )
    (conf5): Sequential(
      (conv_conf5): Conv2d(256, 84, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf5): ReLU()
    )
    (conf6): Sequential(
      (conv_conf6): Conv2d(256, 84, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf6): ReLU()
    )
  )
  (predictor): Predictor()
)

My loss function is defined as;

class SSDLoss(nn.Module):
    def __init__(self, alpha=1, matching_func=None, loc_loss=None, conf_loss=None):
        super().__init__()

        self.alpha = alpha
        self.matching_strategy = matching_strategy if matching_func is None else matching_func
        self.loc_loss = LocalizationLoss() if loc_loss is None else loc_loss
        self.conf_loss = ConfidenceLoss() if conf_loss is None else conf_loss

    def forward(self, predicts, gts, dboxes):
        """
        :param predicts: Tensor, shape is (batch, total_dbox_nums, 4+class_nums=(cx, cy, w, h, p_class,...)
        :param gts: Tensor, shape is (batch*bbox_nums(batch), 1+4+class_nums) = [[img's_ind, cx, cy, w, h, p_class,...],..
        :param dboxes: Tensor, shape is (total_dbox_nums, 4=(cx,cy,w,h))
        :return:
            loss: float
        """
        # get predict's localization and confidence
        pred_loc, pred_conf = predicts[:, :, :4], predicts[:, :, 4:]

        # matching
        pos_indicator, gt_loc, gt_conf = self.matching_strategy(gts, dboxes, batch_num=predicts.shape[0], threshold=0.5)

        # calculate ground truth value considering default boxes
        gt_loc = gt_loc_converter(gt_loc, dboxes)

        # Localization loss
        loc_loss = self.loc_loss(pos_indicator, pred_loc, gt_loc)

        # Confidence loss
        conf_loss = self.conf_loss(pos_indicator, pred_conf, gt_conf)

        return conf_loss + self.alpha * loc_loss

class LocalizationLoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.smoothL1Loss = nn.SmoothL1Loss(reduction='none')

    def forward(self, pos_indicator, predicts, gts):
        N = pos_indicator.sum()

        total_loss = self.smoothL1Loss(predicts, gts).sum(dim=-1) # shape = (batch num, dboxes num)
        loss = total_loss.masked_select(pos_indicator)

        return loss.sum() / N

class ConfidenceLoss(nn.Module):
    def __init__(self, neg_factor=3):
        """
        :param neg_factor: int, the ratio(1(pos): neg_factor) to learn pos and neg for hard negative mining
        """
        super().__init__()
        self.logsoftmax = nn.LogSoftmax(dim=-1)
        self._neg_factor = neg_factor

    def forward(self, pos_indicator, predicts, gts):

        loss = (-gts * self.logsoftmax(predicts)).sum(dim=-1) # shape = (batch num, dboxes num)

        N = pos_indicator.sum()
        neg_indicator = torch.logical_not(pos_indicator)

        pos_loss = loss.masked_select(pos_indicator)
        neg_loss = loss.masked_select(neg_indicator)

        neg_num = neg_loss.shape[0]
        neg_num = min(neg_num, self._neg_factor * N)

        _, topk_indices = torch.topk(neg_loss, neg_num)
        neg_loss = neg_loss.index_select(dim=0, index=topk_indices)

        return (pos_loss.sum() + neg_loss.sum()) / N

loss output is below;

Training... Epoch: 1, Iter: 1,   [32/21503   (0%)]  Loss: 28.804445
Training... Epoch: 1, Iter: 10,  [320/21503  (1%)]  Loss: 12.880742
Training... Epoch: 1, Iter: 20,  [640/21503  (3%)]  Loss: 15.932519
Training... Epoch: 1, Iter: 30,  [960/21503  (4%)]  Loss: 14.624641
Training... Epoch: 1, Iter: 40,  [1280/21503     (6%)]  Loss: 16.301014
Training... Epoch: 1, Iter: 50,  [1600/21503     (7%)]  Loss: 15.710087
Training... Epoch: 1, Iter: 60,  [1920/21503     (9%)]  Loss: 12.441727
Training... Epoch: 1, Iter: 70,  [2240/21503     (10%)] Loss: 12.283393
Training... Epoch: 1, Iter: 80,  [2560/21503     (12%)] Loss: 12.272835
Training... Epoch: 1, Iter: 90,  [2880/21503     (13%)] Loss: 12.273635
Training... Epoch: 1, Iter: 100,     [3200/21503     (15%)] Loss: 12.273409
Training... Epoch: 1, Iter: 110,     [3520/21503     (16%)] Loss: 12.266172
Training... Epoch: 1, Iter: 120,     [3840/21503     (18%)] Loss: 12.272820
Training... Epoch: 1, Iter: 130,     [4160/21503     (19%)] Loss: 12.274920
Training... Epoch: 1, Iter: 140,     [4480/21503     (21%)] Loss: 12.275247
Training... Epoch: 1, Iter: 150,     [4800/21503     (22%)] Loss: 12.273258
Training... Epoch: 1, Iter: 160,     [5120/21503     (24%)] Loss: 12.277486
Training... Epoch: 1, Iter: 170,     [5440/21503     (25%)] Loss: 12.266512
Training... Epoch: 1, Iter: 180,     [5760/21503     (27%)] Loss: 12.265674
Training... Epoch: 1, Iter: 190,     [6080/21503     (28%)] Loss: 12.265306
Training... Epoch: 1, Iter: 200,     [6400/21503     (30%)] Loss: 12.269717
Training... Epoch: 1, Iter: 210,     [6720/21503     (31%)] Loss: 12.274122
Training... Epoch: 1, Iter: 220,     [7040/21503     (33%)] Loss: 12.263970
Training... Epoch: 1, Iter: 230,     [7360/21503     (34%)] Loss: 12.267252

Output:
I must normalize predicted boxes before calculating loss function.

The word of variance caused to mislead...
link

class Encoder(nn.Module):
    def __init__(self, norm_means=(0, 0, 0, 0), norm_stds=(0.1, 0.1, 0.2, 0.2)):
        super().__init__()
        # shape = (1, 1, 4=(cx, cy, w, h)) or (1, 1, 1)
        self.norm_means = torch.tensor(norm_means, requires_grad=False).unsqueeze(0).unsqueeze(0)
        self.norm_stds = torch.tensor(norm_stds, requires_grad=False).unsqueeze(0).unsqueeze(0)

    def forward(self, gt_boxes, default_boxes):
        """
        :param gt_boxes: Tensor, shape = (batch, default boxes num, 4)
        :param default_boxes: Tensor, shape = (default boxes num, 4)
        Note that 4 means (cx, cy, w, h)
        :return:
            encoded_boxes: Tensor, calculate ground truth value considering default boxes. The formula is below;
                           gt_cx = (gt_cx - dbox_cx)/dbox_w, gt_cy = (gt_cy - dbox_cy)/dbox_h,
                           gt_w = train(gt_w / dbox_w), gt_h = train(gt_h / dbox_h)
                           shape = (batch, default boxes num, 4)
        """
        assert gt_boxes.shape[1:] == default_boxes.shape, "gt_boxes and default_boxes must be same shape"

        gt_cx = (gt_boxes[:, :, 0] - default_boxes[:, 0]) / default_boxes[:, 2]
        gt_cy = (gt_boxes[:, :, 1] - default_boxes[:, 1]) / default_boxes[:, 3]
        gt_w = torch.log(gt_boxes[:, :, 2] / default_boxes[:, 2])
        gt_h = torch.log(gt_boxes[:, :, 3] / default_boxes[:, 3])

        encoded_boxes = torch.cat((gt_cx.unsqueeze(2),
                          gt_cy.unsqueeze(2),
                          gt_w.unsqueeze(2),
                          gt_h.unsqueeze(2)), dim=2)

        # normalization
        return (encoded_boxes - self.norm_means.to(gt_boxes.device)) / self.norm_stds.to(gt_boxes.device) <<<<<<<<<<<<< answer!!


Instruction: is crossentropy loss of pytorch different than "categorical_crossentropy" of keras?

Input:
I am trying to mimic a pytorch neural network in keras.

I am confident that my keras version of the neural network is very close to the one in pytorch but during training, I see that the loss value of the pytorch network are much lower than the loss values of the keras network. I wonder if this is because I have not properly copied the pytorch network in keras or the loss computation is different in the two framework.

Pytorch loss definition:

loss_function = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=0.9, weight_decay=5e-4)

Keras loss definition:

sgd = optimizers.SGD(lr=.1, momentum=0.9, nesterov=True)
resnet.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['categorical_accuracy'])

Note that all the layers in the keras network have been implemented with L2 regularization kernel_regularizer=regularizers.l2(5e-4), also I used he_uniform initialization which I believe is default in pytorch, according to the source code.

The batch size for the two networks are the same: 128.

In the pytorch version, I get loss values around 4.1209 which decreases to around 0.5. In keras it starts around 30 and decreases to 2.5.

Output:
PyTorch CrossEntropyLoss accepts unnormalized scores for each class i.e., not probability (source). Keras categorical_crossentropy by default uses from_logits=False which means it assumes y_pred contains probabilities (not raw scores) (source).
In PyTorch, if you use CrossEntropyLoss, you should not use the softmax/sigmoid layer at the end. In keras you can use it or not use it but set the from_logits accordingly.


Instruction: Add tensor to another by repeating along axis

Input:
I have a tensor out with shape:

torch.Size([4, 644, 25])

and another one x with shape:

torch.Size([4, 161, 25])

I want to add x to out 4 times, something like:

        out[:, 0:161] += x
        out[:, 161:322] += x
        out[:, 322:483] += x
        out[:, 483:644] += x

Is there some one-liner that I can use to do this?

Output:
We can use np.tile here:

out += np.tile(x, (1,out.shape[1]//x.shape[1],1))

Or using pytorch's repeat:

out += x.repeat(1,out.shape[1]//x.shape[1],1)


Instruction: Pytorch: How to optimize multiple variables with respect to multiple losses?

Input:
I want different losses to have their gradients computed with respect to different variables, and those variables to then all step together.

Here's a simple example demonstrating what I want:

import torch as T
x = T.randn(3, requires_grad = True)
y = T.randn(4, requires_grad = True)
z = T.randn(5, requires_grad = True)

x_opt = T.optim.Adadelta([x])
y_opt = T.optim.Adadelta([y])
z_opt = T.optim.Adadelta([z])

for i in range(n_iter):
  x_opt.zero_grad()
  y_opt.zero_grad()
  z_opt.zero_grad()

  shared_computation = foobar(x, y, z)

  x_loss = f(x, y, z, shared_computation)
  y_loss = g(x, y, z, shared_computation)
  z_loss = h(x, y, z, shared_computation)

  x_loss.backward_with_respect_to(x)
  y_loss.backward_with_respect_to(y)
  z_loss.backward_with_respect_to(z)

  x_opt.step()
  y_opt.step()
  z_opt.step()

My question is how do we do that backward_with_respect_to part in PyTorch? I only want x's gradient w.r.t. x_loss, etc.. And then I want all the optimizers to step together (based on the current values of x, y, and z).

Output:
I've written a function to do just this. The two key components are (1) using retain_graph=True for all but the final call to .backward() and (2) saving grads after each call to .backward(), and restoring them at the end before .step()ing.

def multi_step(losses, optms):
  # optimizers each take a step, with `optms[i]`'s variables being
  # optimized w.r.t. `losses[i]`.
  grads = [None]*len(losses)
  for i, (loss, optm) in enumerate(zip(losses, optms)):
    retain_graph = i != (len(losses)-1)
    optm.zero_grad()
    loss.backward(retain_graph=retain_graph)
    grads[i] = [
          [
            p.grad+0 for p in group['params']
          ] for group in optm.param_groups
        ]
  for optm, grad in zip(optms, grads):
    for p_group, g_group in zip(optm.param_groups, grad):
      for p, g in zip(p_group['params'], g_group):
        p.grad = g
    optm.step()

In the example code stated in the question, multi_step would be used as follows:

for i in range(n_iter):
  shared_computation = foobar(x, y, z)

  x_loss = f(x, y, z, shared_computation)
  y_loss = g(x, y, z, shared_computation)
  z_loss = h(x, y, z, shared_computation)

  multi_step([x_loss, y_loss, z_loss], [x_opt, y_opt, z_opt])


Instruction: PyTorch LSTM crashing on colab gpu (works fine on cpu)

Input:
Hello I have following LSTM which runs fine on a CPU.

import torch

class LSTMForecast(torch.nn.Module):
    """
    A very simple baseline LSTM model that returns
    an output sequence given a multi-dimensional input seq. Inspired by the StackOverflow link below.
    https://stackoverflow.com/questions/56858924/multivariate-input-lstm-in-pytorch
    """
    def __init__(self, seq_length: int, n_time_series: int, output_seq_len=1, hidden_states:int=20, num_layers=2, bias=True, batch_size=100):
        super().__init__()
        self.forecast_history = seq_length
        self.n_time_series = n_time_series
        self.hidden_dim = hidden_states
        self.num_layers = num_layers
        self.lstm = torch.nn.LSTM(n_time_series, hidden_states, num_layers, bias, batch_first=True)
        self.final_layer = torch.nn.Linear(seq_length*hidden_states, output_seq_len)
        self.init_hidden(batch_size)

    def init_hidden(self, batch_size)->None:
        # This is what we'll initialise our hidden state
        self.hidden = (torch.zeros(self.num_layers, batch_size, self.hidden_dim), torch.zeros(self.num_layers, batch_size, self.hidden_dim))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        batch_size = x.size()[0]
        self.init_hidden(batch_size)
        out_x, self.hidden = self.lstm(x, self.hidden)
        x = self.final_layer(out_x.contiguous().view(batch_size, -1))
        return x

However, when I try to run on colab GPU it crashes without even an error message.

   model = LSTMForecast(1, 1, batch_size=1).to('cuda')
   a = torch.rand(1, 1, 1).to('cuda')
   model(a)

The logs don't tell me anything either. I'm really at a loss.

Output:
I had to explicitly call CUDA. Once I did that it worked.

def init_hidden(self, batch_size)->None:
    # This is what we'll initialise our hidden state
    self.hidden = (torch.zeros(self.num_layers, batch_size, self.hidden_dim).to('cuda'), torch.zeros(self.num_layers, batch_size, self.hidden_dim).to('cuda'))


Instruction: How can I calculate accuracy for keypoints detection CNN model in pytorch?

Input:
Can someone help me with this please,

def train_net(n_epochs):
    valid_loss_min = np.Inf
    history = {'train_loss': [], 'valid_loss': [], 'epoch': []}

    for epoch in range(n_epochs):
        train_loss = 0.0
        valid_loss = 0.0
        net.train()
        running_loss = 0.0
        for batch_i, data in enumerate(train_loader):
            images = data['image']
            key_pts = data['keypoints']
            key_pts = key_pts.view(key_pts.size(0), -1)
            key_pts = key_pts.type(torch.FloatTensor).to(device)
            images = images.type(torch.FloatTensor).to(device)
            output_pts = net(images)
            loss = criterion(output_pts, key_pts)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            train_loss += loss.item()*images.data.size(0)
        net.eval()

        with torch.no_grad():
            for batch_i, data in enumerate(test_loader):
                images = data['image']
                key_pts = data['keypoints']
                key_pts = key_pts.view(key_pts.size(0), -1)
                key_pts = key_pts.type(torch.FloatTensor).to(device)
                images = images.type(torch.FloatTensor).to(device)
                output_pts = net(images)
                loss = criterion(output_pts, key_pts)
                valid_loss += loss.item()*images.data.size(0)
        train_loss = train_loss/len(train_loader.dataset)
        valid_loss = valid_loss/len(test_loader.dataset)
        print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch+1,train_loss,valid_loss))

        if valid_loss <= valid_loss_min:
            print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,valid_loss))
            torch.save(net,f'X:\\xxxx\\xxx\\xxx\\epoch{epoch + 1}_loss{valid_loss}.pth')
            valid_loss_min = valid_loss
        history['epoch'].append(epoch + 1)
        history['train_loss'].append(train_loss)
        history['valid_loss'].append(valid_loss)
    print('Finished Training')
    return history
'''

Above is the training code for reference!

Output:
This is funny, I was just working on this minutes ago myself! As you probably realise, simply calculating the Euclidean distance between 2 sets of keypoints doesn't generalise well to cases where you need to compare across body shapes and sizes. So I would recommend using the Object Keypoint Similarity Score, which measures the body joints distance normalised by the scale of the person. As represented in this blog, OKS is defined as:

Here (line 313 function computeOKS) is Facebook research's implementation:

def computeOks(self, imgId, catId):
    p = self.params
    # dimention here should be Nxm
    gts = self._gts[imgId, catId]
    dts = self._dts[imgId, catId]
    inds = np.argsort([-d['score'] for d in dts], kind='mergesort')
    dts = [dts[i] for i in inds]
    if len(dts) > p.maxDets[-1]:
        dts = dts[0:p.maxDets[-1]]
    # if len(gts) == 0 and len(dts) == 0:
    if len(gts) == 0 or len(dts) == 0:
        return []
    ious = np.zeros((len(dts), len(gts)))
    sigmas = np.array([.26, .25, .25, .35, .35, .79, .79, .72, .72, .62,.62, 1.07, 1.07, .87, .87, .89, .89])/10.0
    vars = (sigmas * 2)**2
    k = len(sigmas)
    # compute oks between each detection and ground truth object
    for j, gt in enumerate(gts):
        # create bounds for ignore regions(double the gt bbox)
        g = np.array(gt['keypoints'])
        xg = g[0::3]; yg = g[1::3]; vg = g[2::3]
        k1 = np.count_nonzero(vg > 0)
        bb = gt['bbox']
        x0 = bb[0] - bb[2]; x1 = bb[0] + bb[2] * 2
        y0 = bb[1] - bb[3]; y1 = bb[1] + bb[3] * 2
        for i, dt in enumerate(dts):
            d = np.array(dt['keypoints'])
            xd = d[0::3]; yd = d[1::3]
            if k1>0:
                # measure the per-keypoint distance if keypoints visible
                dx = xd - xg
                dy = yd - yg
            else:
                # measure minimum distance to keypoints in (x0,y0) & (x1,y1)
                z = np.zeros((k))
                dx = np.max((z, x0-xd), axis=0) + np.max((z, xd-x1), axis=0)
                dy = np.max((z, y0-yd), axis=0) + np.max((z, yd-y1), axis=0)
            e = (dx**2 + dy**2) / vars / (gt['area'] + np.spacing(1)) / 2
            if k1 > 0:
                e=e[vg > 0]
            ious[i, j] = np.sum(np.exp(-e)) / e.shape[0]
    return ious


Instruction: convert sess.run to pytorch

Input:
I am trying to convert a code from tf to pytorch.
The part of the code where I am stuck is this sess.run.  As fas as I know, pytorch doesn't need it, but I don't find the way to replicate it. I attach you the code.

TF:

ebnos_db = np.linspace(1,6, 6)
bers_no_training = np.zeros(shape=[ebnos_db.shape[0]])
for j in range(epochs):
    for i in range(ebnos_db.shape[0]):
        ebno_db = ebnos_db[i]
        bers_no_training[i] += sess.run(ber, feed_dict={
            batch_size: samples,
            noise_var: ebnodb2noisevar(ebno_db, coderate)
        })
bers_no_training /= epochs

samples is a int32 and ebnodb2noisevar() returns a float32.

BER in TF is calculated as:

ber = tf.reduce_mean(tf.cast(tf.not_equal(x, x_hat), dtype=tf.float32))

and in PT:

wrong_bits = ( torch.eq(x, x_hat).type(torch.float32) * -1 ) + 1
ber = torch.mean(wrong_bits)

I think BER is well computed, but the main problem is that I don't know how to convert sess.run into PyTorch, nor I completely understand its function.

Can anybody help me?

Thanks

Output:
You can do the same in PyTorch but easier when it comes to ber:

ber = torch.mean((x != x_hat).float())

would be enough.

Yes, PyTorch doesn't need it as it's based on dynamic graph construction (unlike Tensorflow with it's static approach).

In tensorflow sess.run is used to feed values into created graph; here tf.Placeholder (variable in graph which represents a node where a user can "inject" his data) named batch_size will be fed with samples and noise_var with ebnodb2noisevar(ebno_db, coderate).

Translating this to PyTorch is usually straightforward as you don't need any graph-like approaches with session. Just use your neural network (or a-like) with correct input (like samples and noise_var) and you are fine. You have to check your graph (so how ber is constructed from batch_size and noise_var) and reimplement it in PyTorch.

Also, please check PyTorch introductory tutorials to get a feel of the framework before diving into it.


Instruction: How to use a Batchsampler within a Dataloader

Input:
I have a need to use a BatchSampler within a pytorch DataLoader instead of calling __getitem__ of the dataset multiple times (remote dataset, each query is pricy). I cannot understand how to use the batchsampler with any given dataset.

e.g

class MyDataset(Dataset):

    def __init__(self, remote_ddf, ):
        self.ddf = remote_ddf

    def __len__(self):
        return len(self.ddf)

    def __getitem__(self, idx):
        return self.ddf[idx] --------> This is as expensive as a batch call

    def get_batch(self, batch_idx):
        return self.ddf[batch_idx]

my_loader = DataLoader(MyDataset(remote_ddf),
           batch_sampler=BatchSampler(Sampler(), batch_size=3))

The thing I do not understand, neither found any example online or in torch docs, is how do I use my get_batch function instead of the __getitem__ function.

Edit:
Following the answer of Szymon Maszke, this is what I tried and yet, \_\_get_item__ gets one index each call, instead of a list of size batch_size

class Dataset(Dataset):

    def __init__(self):
       ...

    def __len__(self):
        ...

    def __getitem__(self, batch_idx):  ------> here I get only one index
        return self.wiki_df.loc[batch_idx]

loader = DataLoader(
                dataset=dataset,
                batch_sampler=BatchSampler(
                    SequentialSampler(dataset), batch_size=self.hparams.batch_size, drop_last=False),
                num_workers=self.hparams.num_data_workers,
            )

Output:
You can't use get_batch instead of __getitem__ and I don't see a point to do it like that.

torch.utils.data.BatchSampler takes indices from your Sampler() instance (in this case 3 of them) and returns it as list so those can be used in your MyDataset __getitem__ method (check source code, most of samplers and data-related utilities are easy to follow in case you need it).

I assume your self.ddf supports list slicing (e.g. self.ddf[[25, 44, 115]] returns values correctly and uses only one expensive call). In this case simply switch get_batch into __getitem__ and you are good to go.

class MyDataset(Dataset):

    def __init__(self, remote_ddf, ):
        self.ddf = remote_ddf

    def __len__(self):
        return len(self.ddf)

    def __getitem__(self, batch_idx):
        return self.ddf[batch_idx] -> batch_idx is a list

EDIT: You have to specify batch_sampler as sampler, otherwise the batch will be divided into single indices. This should be fine:

loader = DataLoader(
    dataset=dataset,
    # This line below!
    sampler=BatchSampler(
        SequentialSampler(dataset), batch_size=self.hparams.batch_size, drop_last=False
    ),
    num_workers=self.hparams.num_data_workers,
)


Instruction: Minimal (light version) PyTorch and Numpy packages in production

Input:
I am putting a model into production and I am required to scan all dependencies (Pytorch and Numpy) beforehand via VeraCode Scan.

I noticed that the majority of the flaws are coming from test scripts and caffe2 modules in Pytorch and numpy.

Is there any way to build/install only part of these packages that I use in my application? (e.g. I won't use testing and caffe2 in the application so there's no need to have them in my PyTorch / Numpy source code)

Output:
1. PyInstaller

You could package your application using pyinstaller. This tool packages your app with Python and dependencies and use only the parts you need (simplifying, in reality it's hard to trace your package exactly so some other stuff would be bundled as well).

Also you might be in for some quirks and workarounds to make it work with pytorch and numpy as those dependencies are quite heavy (especially pytorch).

2. Use only PyTorch

numpy and pytorch are pretty similar feature-wise (as PyTorch tries to be compatible with it) hence maybe you could only use only of them which would simplify the whole thing further

3. Use C++

Depending on other parts of your app you may write it (at least neural network) in C++ using PyTorch's C++ frontend which is stable since 1.5.0 release.

Going this route would allow you to compile PyTorch's .cpp source code statically (so all dependencies are linked) which allows you for relatively small binary size (30Mb when compared to PyTorch's 1GB+), but requires a lot of work.


Instruction: persistent pip install in rapids.ai docker container

Input: This is probably a really stupid question, but one has got to start somewhere. I am playing with NVDIA's rapids.ai gpu-enhanced docker container, but this (presumably by design) does not come with pytorch. Now, of course, I can do a pip install torch torch-ignite every time, but this is both annoying and resource-consuming (and pytorch is a large download). What is the approved method for persisting a pip install in a container?

Output:
Create a new Dockerfile that builds a new image based on the existing one:

FROM the/rapids-ai/image

RUN pip install torch torch-ignite

And then

$ ls Dockerfile
Dockerfile
$ docker build -t myimage .

You can now do:

$ docker run myimage


Instruction: How to keep model fixed during training?

Input:
I am trying to implement a model that uses encoding from multiple pre-trained BERT models on different datasets and gets a combined representation using a fully-connected layer. In this, I want that BERT models should remain fixed and only fully-connected layers should get trained. Is it possible to achieve this in huggingface-transformers? I don't see any flag which allows me to do that.

PS: I don't want to go by the way of dumping the encoding of inputs for each BERT model and use them as inputs.

Output:
A simple solution to this is to just exclude the parameters related to the BERT model while passing to the optimizer.

param_optimizer = [x for x in param_optimizer if 'bert' not in x[0]]
optimizer = AdamW(param_optimizer, lr)


Instruction: Pytorch: looking for a function that let me to manually set learning rates for specific epochs intervals

Input: For example, set lr = 0.01 for the first 100 epochs, lr = 0.001 from epoch 101 to epoch 1000, lr = 0.0005 for epoch 1001-4000. Basically my learning rate plan is not letting it decay exponentially with a fixed number of steps. I know it can be achieved by self-defined functions, just curious if there are already developed functions to do that.

Output:
torch.optim.lr_scheduler.LambdaLR is what you are looking for. It returns multiplier of initial learning rate so you can specify any value for any given epoch. For your example it would be:

def lr_lambda(epoch: int):
    if 100 < epoch < 1000:
        return 0.1
    if 1000 < epoch 4000:
        return 0.05

# Optimizer has lr set to 0.01
scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
for epoch in range(100):
    train(...)
    validate(...)
    optimizer.step()
    scheduler.step()

In PyTorch there are common functions (like MultiStepLR or ExponentialLR) but for custom use case (as is yours), LambdaLR is the easiest.


Instruction: Remove RELU activation from Resnet model in Pytorch

Input: How to remove all RELU activation layers from Resnet model in pytorch and if possible replace it by a linear activation layer?

Output:
model = models.resnet50()

names = []
for name, module in model.named_modules():
    if hasattr(module, 'relu'):
        module.relu = nn.Sigmoid() // or nn.Identity() accordingly

print(model)

This works for either replacing activations or making it identity


Instruction: Python matplotlib, invalid shape for image data

Input:
Currently I have this code to show three images:

imshow(image1, title='1')
imshow(image2, title='2')
imshow(image3, title='3')

And it works fine. But I am trying to put them all three in a row instead of column.

Here is the code I have tried:

f = plt.figure()
f.add_subplot(1,3,1)
plt.imshow(image1)
f.add_subplot(1,3,2)
plt.imshow(image2)
f.add_subplot(1,3,3)
plt.imshow(image3)

It throws

  TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

If I do

f = plt.figure()
f.add_subplot(1,3,1)
plt.imshow(image1.cpu())
f.add_subplot(1,3,2)
plt.imshow(image2.cpu())
f.add_subplot(1,3,3)
plt.imshow(image3.cpu())

It throws

  TypeError: Invalid shape (1, 3, 128, 128) for image data

How should I fix this or is there an easier way to implement it?

Output:
The matplotlib function 'imshow' gets 3-channel pictures as (h, w, 3) as you can see in the documentation.

It seems that you passed a "batch" of single image (the first dimention) of three channels (second dimention) of the image (h and w are the third and forth dimention).

You need to reshape or view your image (after converting to cpu, try to use:

image1.squeeze().permute(1,2,0)

The result will be an image of the desired shape (128, 128, 3).

The squeeze() function will remove the first dimention. And the premute() function will transpose the dimenstion where the first will shift to the third position and the two other will shift to the beginning.

Also, have a look here for further talk on the GPU and CPU issues:
link

Hope that helps.


Instruction: I am trying to use pytorch's implementation of XLNet and got 'Trying to create tensor with negative dimension -1: [-1, 768]' when loading XLNet

Input:
I started working on this about two months ago on Google Colab for a midterm project and everything worked perfectly. Now I am modifying it for a final project and keep getting the error 'RuntimeError: Trying to create tensor with negative dimension -1: [-1, 768]'. It looks like pytorch recently pushed a new version 1.5, so I downgraded to version 1.4 and still got the same error. Same with 1.3, and I know I wasn't using anything lower since that came out last year. I checked it with my midterm code and still got the same error, so I don't know what's going on. Here is the chunk of code related to downloading and using the model.

    train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(inputIds,
                                                                                labels,
                                                                                random_state=2020,
                                                                                test_size=0.2)
    train_masks, validation_masks, _, _ = train_test_split(attention_masks, inputIds, random_state=2020,
                                                           test_size=0.2)
    # Turn data into torch tensors
    train_inputs = torch.tensor(train_inputs)
    validation_inputs = torch.tensor(validation_inputs)
    train_labels = torch.tensor(train_labels)
    validation_labels = torch.tensor(validation_labels)
    train_masks = torch.tensor(train_masks)
    validation_masks = torch.tensor(validation_masks)

    # Create Iterators of the datasets
    train_data = TensorDataset(train_inputs, train_masks, train_labels)
    train_sampler = RandomSampler(train_data)
    train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)
    validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)
    validation_sampler = SequentialSampler(validation_data)
    validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)

    model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2)
    # Loads model into GPU memory
    model.cuda()

    param_optimizer = list(model.named_parameters())
    no_decay = ['bias','gamma','beta']
    optimizer_grouped_parameters = [
        {'params':[p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'weight_decay_rate':0.01},
        {'params':[p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'weight_decay_rate':0.0}
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5)

The error happens on the line model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2). The packages I am using:

from pandas import to_datetime
import torch
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler
from keras.preprocessing.sequence import pad_sequences
from sklearn.model_selection import train_test_split
# MUST INSTALL PYTORCH-TRANSFORMERS
from pytorch_transformers import XLNetTokenizer, XLNetForSequenceClassification, AdamW
from tqdm import trange
from numpy import argmax, sum
import nltk
nltk.download('punkt')

Thank you to anyone who tries to help.

Output:
You can try transformers instead of pytorch_transformers.

! pip install transformers (Google Colab)

In terminal,

pip install transformers

import torch

from transformers import XLNetForSequenceClassification

model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2)

model.cuda()

param_optimizer = list(model.named_parameters())
no_decay = ['bias','gamma','beta']
optimizer_grouped_parameters = [
    {'params':[p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
      'weight_decay_rate':0.01},
    {'params':[p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
      'weight_decay_rate':0.0}
]

Here's the code without any error in google colab: https://colab.research.google.com/drive/1A8edGYyFuE7d1Z-ZJusz5zU_7yCevF2L


Instruction: Input dimension for CrossEntropy Loss in PyTorch

Input:
For a binary classification problem with batch_size = 1, I have logit and label values using which I need to calculate loss.

logit: tensor([0.1198, 0.1911], device='cuda:0', grad_fn=<AddBackward0>)
label: tensor(1], device='cuda:0')
# calculate loss
loss_criterion = nn.CrossEntropyLoss()
loss_criterion.cuda()
loss = loss_criterion( b_logits, b_labels )

However, this always results in the following error,

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

What input dimensions is the CrossEntropyLoss actually asking for?

Output:
You are passing wrong shape of tensors.
shape should be (from doc)

  Input: (N,C) where C = number of classes
  Target: (N) where each value is 0 ≤ targets[i] ≤ C−1

So here, b_logits shape should be ([1,2]) instead of ([2]) to make it right shape you can use torch.view like b_logits.view(1,-1).

And b_labels shape should be ([1]).
Ex.:

b_logits = torch.tensor([0.1198, 0.1911], requires_grad=True)
b_labels = torch.tensor([1])
loss_criterion = nn.CrossEntropyLoss()

loss = loss_criterion( b_logits.view(1,-1), b_labels )
loss
tensor(0.6581, grad_fn=<NllLossBackward>)


Instruction: Filter out np.nan values from pytorch 1d tensor

Input:
I have a 1d tensor looking kinda like this:

import numpy as np
import torch

my_list = [0, 1, 2, np.nan, np.nan, 4]
tensor = torch.Tensor(my_list)

How do i filter out the nan-values, so it becomes a tensor of size 4?

Output:
You can use torch.isnan

my_list = [0, 1, 2, np.nan, np.nan, 4]
tensor = torch.Tensor(my_list)

tensor[~torch.isnan(tensor)]
tensor([0., 1., 2., 4.])


Instruction: Error while applying image augmentation transformations to data in FastAI

Input:
I am trying to replicate this kaggle notebook https://www.kaggle.com/tanlikesmath/diabetic-retinopathy-with-resnet50-oversampling on Google Colab. The code was working fine till yesterday but today it is throwing a runtime error. Below is the problematic code:

tfms = get_transforms(do_flip=True,flip_vert=True,max_rotate=360,max_warp=0,max_zoom=1.1,max_lighting=0.1,p_lighting=0.5)
src = (ImageList.from_df(df=df,path=data_path,cols='path') #get dataset from dataset //ImageItemList threw errors so changed to ImageList
        .split_by_idx(range(len(train_df)-1,len(df))) #Splitting the dataset
        .label_from_df(cols='level') #obtain labels from the level column
      )
data= (src.transform(tfms,size=sz) #Data augmentation
        .databunch(bs=bs,num_workers=0) #DataBunch
        .normalize(imagenet_stats) #Normalize
       )

I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in _check_kwargs(ds, tfms, **kwargs)
    593         x = ds[0]
--> 594         try: x.apply_tfms(tfms, **kwargs)
    595         except Exception as e:

8 frames
/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in apply_tfms(self, tfms, do_resolve, xtra, size, resize_method, mult, padding_mode, mode, remove_out)
    122                     x = tfm(x, size=_get_crop_target(size,mult=mult), padding_mode=padding_mode)
--> 123             else: x = tfm(x)
    124         return x.refresh()

/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in __call__(self, x, *args, **kwargs)
    523         "Randomly execute our tfm on `x`."
--> 524         return self.tfm(x, *args, **{**self.resolved, **kwargs}) if self.do_run else x
    525

/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in __call__(self, p, is_random, use_on_y, *args, **kwargs)
    469         "Calc now if `args` passed; else create a transform called prob `p` if `random`."
--> 470         if args: return self.calc(*args, **kwargs)
    471         else: return RandTransform(self, kwargs=kwargs, is_random=is_random, use_on_y=use_on_y, p=p)

/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in calc(self, x, *args, **kwargs)
    474         "Apply to image `x`, wrapping it if necessary."
--> 475         if self._wrap: return getattr(x, self._wrap)(self.func, *args, **kwargs)
    476         else:          return self.func(x, *args, **kwargs)

/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in affine(self, func, *args, **kwargs)
    182         m = tensor(func(*args, **kwargs)).to(self.device)
--> 183         self.affine_mat = self.affine_mat @ m
    184         return self

RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #3 'mat2' in call to _th_addmm_out

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
<ipython-input-74-31aae73a70fc> in <module>()
      6       )
      7 print(src)
----> 8 data= (src.transform(tfms,size=sz) #Data augmentation
      9         .databunch(bs=bs,num_workers=0) #DataBunch
     10         .normalize(imagenet_stats) #Normalize

/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in transform(self, tfms, **kwargs)
    503         if not tfms: tfms=(None,None)
    504         assert is_listy(tfms) and len(tfms) == 2, "Please pass a list of two lists of transforms (train and valid)."
--> 505         self.train.transform(tfms[0], **kwargs)
    506         self.valid.transform(tfms[1], **kwargs)
    507         if self.test: self.test.transform(tfms[1], **kwargs)

/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in transform(self, tfms, tfm_y, **kwargs)
    722     def transform(self, tfms:TfmList, tfm_y:bool=None, **kwargs):
    723         "Set the `tfms` and `tfm_y` value to be applied to the inputs and targets."
--> 724         _check_kwargs(self.x, tfms, **kwargs)
    725         if tfm_y is None: tfm_y = self.tfm_y
    726         tfms_y = None if tfms is None else list(filter(lambda t: getattr(t, 'use_on_y', True), listify(tfms)))

/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in _check_kwargs(ds, tfms, **kwargs)
    594         try: x.apply_tfms(tfms, **kwargs)
    595         except Exception as e:
--> 596             raise Exception(f"It's not possible to apply those transforms to your dataset:\n {e}")
    597
    598 class LabelList(Dataset):

Exception: It's not possible to apply those transforms to your dataset:
 Expected object of scalar type Float but got scalar type Double for argument #3 'mat2' in call to _th_addmm_out

I changed nothing in this code, it is the same as it was yesterday but for some reason it gives me an error today. Kindly Help.

Edit: I found out it is working perfectly fine on my local Jupyter notebook. Still shows error for Colab though

Output:
It seems that some issues with torch that is used in colab

FastAI Forum
Try to install specific version of torch in your colab before run fastAI python code

!pip install "torch==1.4" "torchvision==0.5.0"


Instruction: Best practice to pass PyTorch device name to model

Input:
Currently, I separated train.py with model.py for my deep learning project.

So for the datasets, they are sent to cuda device inside the epoch for loop like below.

train.py

...
device = torch.device('cuda:2' if torch.cuda.is_available() else 'cpu')
model = MyNet(~).to(device)
...
for batch_data in train_loader:
    s0 = batch_data[0].to(device)
    s1 = batch_data[1].to(device)
    pred = model(s0, s1)

However, inside my model (in model.py), it also needs to access the device variable for skip connection like method. To make a new copy of hidden unit (for residual connection)

model.py

class MyNet(nn.Module):
    def __init__(self, in_feats, hid_feats, out_feats):
        super(MyNet, self).__init__()
        self.conv1 = GCNConv(in_feats, hid_feats)
        ...

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x1 = copy.copy(x.float())
        x = self.conv1(x, edge_index)
        skip_conn = torch.zeros(len(data.batch), x1.size(1)).to(device)  # <--
        (some opps for x1 -> skip_conn)
        x = torch.cat((x, skip_conn), 1)

In this case, I am currently passing device as a parameter, however, I believe this is not a best practice.

Where should be the best practice to send the dataset to CUDA?
In the case of multiple scripts need to access device, how sould I handle this? (parameter, global variable?)

Output:
You can add a new attribute to MyModel to store the device info and use this in the skip_conn initialization.

class MyNet(nn.Module):
def __init__(self, in_feats, hid_feats, out_feats, device): # <--
    super(MyNet, self).__init__()
    self.conv1 = GCNConv(in_feats, hid_feats)
    self.device = device # <--
    self.to(self.device) # <--
    ...

def forward(self, data):
    x, edge_index = data.x, data.edge_index
    x1 = copy.copy(x.float())
    x = self.conv1(x, edge_index)
    skip_conn = torch.zeros(len(data.batch), x1.size(1), device=self.device)  # <--
    (some opps for x1 -> skip_conn)
    x = torch.cat((x, skip_conn), 1)

Notice that in this example, MyNet is responsible for all the device logic including the .to(device) call. This way, we are encapsulating all model-related device management in the model class itself.


Instruction: Cross-validation of neural network: How to treat the number of epochs?

Input:
I'm implementing a pytorch neural network (regression) and want to identify the best network topology, optimizer etc.. I use cross validation, because I have x databases of measurements and I want to evaluate whether I can train a neural network with a subset of the x databases and apply the neural network to the unseen databases. Therefore, I also introduce a test database, which I doesn't use in the phase of the hyperparameter identification.
I am confused on how to treat the number of epochs in cross validation, e.g. I have a number of epochs = 100. There are two options:

The number of epochs is a hyperparameter to tune. In each epoch, the mean error across all cross validation iterations is determined. After models are trained with all network topologies, optimizers etc. the model with the smallest mean error is determined and has parameters like: -network topology: 1
-optimizer: SGD
-number of epochs: 54
To calculate the performance on the test set, a model is trained with exactly these parameters (number of epochs = 54) on the training and the validation data. Then it is applied and evaluated on the test set.
The number of epochs is NOT a hyperparameter to tune. Models are trained with all the network topologies, optimizers etc. For each model, the number of epochs, where the error is the smallest, is used. The models are compared and the best model can be determined with parameters like:
-network topology: 1
-optimizer: SGD
To calculate the performance on the test data, a “simple” training and validation split is used (e.g. 80-20). The model is trained with the above parameters and 100 epochs on the training and validation data. Finally, a model with a number of epochs yielding the smallest validation error, is evaluated on the test data.

Which option is the correct or the better one?

Output:
The number of epochs is better not to be fine-tuned.
Option 2 is a better option.
Actually, if the # of epochs is fixed, you need not to have validation set. Validation set gives you the optimal epoch of the saved model.


Instruction: GPU usage shows zero when CUDA with PyTorch using on Windows

Input:
I have pytorch script.

import torch

torch.cuda.is_available()
# True

device=torch.device('cuda:0')
# I moved my tensors to device

But Windows Task Manager shows zero GPU (NVIDIA GTX 1050TI) usage when pytorch script running
Speed of my script is fine and if I had changing torch.device to CPU instead GPU a speed become slower, therefore cuda (GPU) is working. Why Windows Task Manager doesn't show GPU usage?

Sample of my code:

device=torch.device("cuda:0")
model=torch.load('mymodel.pth', map_location=torch.device(device))
image=Image.open('picture.png').convert('RGB')
transform=transforms.Compose([
            transforms.Resize(224),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])
input=transform(image)
input=torch.unsqueeze(input, 0)
input=input.to(device)
output=model(input)

Output:
Windows task manager overall utilization does not seem to include cuda usage. Make sure you select the cuda option in the graphs.
For details see: https://medium.com/@michaelceber/gpu-monitoring-on-windows-10-for-machine-learning-cuda-41088de86d65


Instruction: How does one pickle arbitrary pytorch models that use lambda functions?

Input:
I currently have a neural network module:

import torch.nn as nn

class NN(nn.Module):
    def __init__(self,args,lambda_f,nn1, loss, opt):
        super().__init__()
        self.args = args
        self.lambda_f = lambda_f
        self.nn1 = nn1
        self.loss = loss
        self.opt = opt
        # more nn.Params stuff etc...

    def forward(self, x):
        #some code using fields
        return out

I am trying to checkpoint it but because pytorch saves using state_dicts it means I can't save the lambda functions I was actually using if I checkpoint with the pytorch torch.save etc. I literally want to save everything without issue and re-load to train on GPUs later. I currently am using this:

def save_ckpt(path_to_ckpt):
    from pathlib import Path
    import dill as pickle
    ## Make dir. Throw no exceptions if it already exists
    path_to_ckpt.mkdir(parents=True, exist_ok=True)
    ckpt_path_plus_path = path_to_ckpt / Path('db')

    ## Pickle args
    db['crazy_mdl'] = crazy_mdl
    with open(ckpt_path_plus_path , 'ab') as db_file:
        pickle.dump(db, db_file)

currently it throws no errors when I chekpoint it and it saved it.

I am worried that when I train it there might be a subtle bug even if no exceptions/errors are trained or something unexpected might happen (e.g. weird saving on disks in the clusters etc who knows).

Is this safe to do with pytorch classes/nn models? Especially if we want to resume training with GPUs?

Cross posted:

How does one pickle arbitrary pytorch models that use lambda functions?
https://discuss.pytorch.org/t/how-does-one-pickle-arbitrary-pytorch-models-that-use-lambda-functions/79026
https://www.reddit.com/r/pytorch/comments/gagpjg/how_does_one_pickle_arbitrary_pytorch_models_that/?
https://www.quora.com/unanswered/How-does-one-pickle-arbitrary-PyTorch-models-that-use-lambda-functions

Output: this is not a good idea. If you do this then if your code changes to a different github repo then it will be hard restore your models that took a lot of time to train. The cycles spent recovering those or retraining is not worth it. I recommend to instead do it the pytorch way and only save the weights as they recommend in pytorch.


Instruction: How is Siamese network realized with Pytorch if it is single input during inference?

Input:
I am trying to train one CNN model with Pytorch, so that the output behaves differently for different types of inputs. (i.e. If the input images are human-beings, it outputs pattern A, but if the input is some other animals, it outputs pattern B).

After some online search, it seems Siamese network is related to this. So I have the following 2 questions:

(1)  Is Siamese network really a good way to train such a model?

(2)  From the implementation point of view, how should I implement the code in pytorch?

class SiameseNetwork(nn.Module):
def __init__(self):
    super(SiameseNetwork, self).__init__()
    self.cnn1 = nn.Sequential(
        nn.ReflectionPad2d(1),
        nn.Conv2d(1, 4, kernel_size=3),
        nn.ReLU(inplace=True),
        nn.BatchNorm2d(4),

        nn.ReflectionPad2d(1),
        nn.Conv2d(4, 8, kernel_size=3),
        nn.ReLU(inplace=True),
        nn.BatchNorm2d(8),

        nn.ReflectionPad2d(1),
        nn.Conv2d(8, 8, kernel_size=3),
        nn.ReLU(inplace=True),
        nn.BatchNorm2d(8),

    )

    self.fc1 = nn.Sequential(
        nn.Linear(8*100*100, 500),
        nn.ReLU(inplace=True),

        nn.Linear(500, 500),
        nn.ReLU(inplace=True),

        nn.Linear(500, 5))

def forward_once(self, x):
    output = self.cnn1(x)
    output = output.view(output.size()[0], -1)
    output = self.fc1(output)
    return output

def forward(self, input1, input2):
    output1 = self.forward_once(input1)
    output2 = self.forward_once(input2)
    return output1, output2

Currently, I am trying some existing implementation I found online like the above class definition. It works, but there will always be two inputs and two outputs for this model. I agree that it is convenient for training, but ideally, it should be only one input and one (two is also fine) output during inference.

Could someone provide some guidance on how to modify the code to make it single input?

Output:
You can call forward_once during inference: this takes a single input and returns a single output. Note that explicitly calling forward_once will not invoke any hooks you might have on forward/backward calls of your module.

Alternatively, you can make forward_once your module's forward function, and make your training function do the double calling of your model (which makes more sense: Siamese networks is a training method, and not part of a network's architecture).


Instruction: How to concatenate a tensor WITHIN axis=1?

Input:
I have a tensor of shape(2,2,2,2):

tensor([[[[   5.,    5.],
          [   5.,    5.]],

         [[  10.,   10.],
          [  10.,   10.]]],

        [[[ 100.,  100.],
          [ 100.,  100.]],

         [[1000., 1000.],
          [1000., 1000.]]]], device='cuda:0')

I want to transform it such that the tensor along axis=1 are repeated 3 times. And after applying .view(-1) to that I get a 1D resultant tensor as:

tensor([   5.,    5.,    5.,    5.,   5.,    5.,    5.,    5.,   5.,    5.,    5.,    5.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,  100.,  100.,  100.,  100.,  100.,  100., 100.,  100.,  100.,  100., 100.,  100.,  100.,  100.,  100.,  100., 1000., 1000., 1000., 1000.  1000., 1000., 1000., 1000. 1000., 1000., 1000., 1000. 1000., 1000., 1000., 1000.], device='cuda:0')

How to do this?

Output:
Use torch.repeat_interleave to Repeat elements of a tensor.

t.repeat_interleave(repeats=3, dim=1).view(-1)
tensor([   5.,    5.,    5.,    5.,    5.,    5.,    5.,    5.,    5.,    5.,
           5.,    5.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,
          10.,   10.,   10.,   10.,  100.,  100.,  100.,  100.,  100.,  100.,
         100.,  100.,  100.,  100.,  100.,  100., 1000., 1000., 1000., 1000.,
        1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000.])


Instruction: Custom submodules in pytorch / libtorch C++

Input:
Full disclosure, I asked this same question on the PyTorch forums about a few days ago and got no reply, so this is technically a repost, but I believe it's still a good question, because I've been unable to find an answer anywhere online. Here goes:

Can you show an example of using register_module with a custom module?
The only examples I’ve found online are registering linear layers or convolutional layers as the submodules.

I tried to write my own module and register it with another module and I couldn’t get it to work.
My IDE is telling me no instance of overloaded function "MyModel::register_module" matches the argument list -- argument types are: (const char [14], TreeEmbedding)

(TreeEmbedding is the name of another struct I made which extends torch::nn::Module.)

Am I missing something? An example of this would be very helpful.

Edit: Additional context follows below.

I have a header file "model.h" which contains the following:

struct TreeEmbedding : torch::nn::Module {
    TreeEmbedding();
    torch::Tensor forward(Graph tree);
};

struct MyModel : torch::nn::Module{
    size_t embeddingSize;
    TreeEmbedding treeEmbedding;

    MyModel(size_t embeddingSize=10);
    torch::Tensor forward(std::vector<Graph> clauses, std::vector<Graph> contexts);
};

I also have a cpp file "model.cpp" which contains the following:

MyModel::MyModel(size_t embeddingSize) :
    embeddingSize(embeddingSize)
{
    treeEmbedding = register_module("treeEmbedding", TreeEmbedding{});
}

This setup still has the same error as above. The code in the documentation does work (using built-in components like linear layers), but using a custom module does not. After tracking down torch::nn::Linear, it looks as though that is a ModuleHolder (Whatever that is...)

Thanks,
Jack

Output:
I will accept a better answer if anyone can provide more details, but just in case anyone's wondering, I thought I would put up the little information I was able to find:

register_module takes in a string as its first argument and its second argument can either be a ModuleHolder (I don't know what this is...) or alternatively it can be a shared_ptr to your module. So here's my example:

treeEmbedding = register_module<TreeEmbedding>("treeEmbedding", make_shared<TreeEmbedding>());

This seemed to work for me so far.


Instruction: Target size (torch.Size([12])) must be the same as input size (torch.Size([12, 1000]))

Input:
I am using models.vgg16(pretrained=True) model for image classification, where number of classes = 3.

Batch size is 12 trainloader = torch.utils.data.DataLoader(train_data, batch_size=12, shuffle=True) since error says Target size (torch.Size([12])) must be the same as input size (torch.Size([12, 1000]))

I have changed last fc layer parameters and got last FC layer as  Linear(in_features=1000, out_features=3, bias=True)

Loss function is BCEWithLogitsLoss()

criterion = nn.BCEWithLogitsLoss()
optimizer = optim.SGD(vgg16.parameters(), lr=0.001, momentum=0.9)

Training code is

        # zero the parameter gradients
        optimizer.zero_grad()
        outputs = vgg16(inputs)               #----> forward pass
        loss = criterion(outputs, labels)   #----> compute loss #error occurs here
        loss.backward()                     #----> backward pass
        optimizer.step()                    #----> weights update

While computing loss, I get this error Target size (torch.Size([12])) must be the same as input size (torch.Size([12, 1000]))

code is available at: code

Output:
Try to double check how you modified the linear layer. It seems that somehow the model does not forward pass through it.

Your model output have 1000 output size for each sample, while it should have 3. That's the reason you cannot evaluate the loss, since you try to compare 1000 classes to 3. You should have 3 outputs in your last layer, and that should work.

EDIT

From the code you shared here: link, I think there are two problems.

First, you modifed your model this way:

# Load the pretrained model from pytorch
vgg16 = models.vgg16(pretrained=True)

vgg16.classifier[6].in_features = 1000
vgg16.classifier[6].out_features = 3

while what you did here is to add a layer as an attribute to your network, you should also modify the forward() function of your model. Adding the layer as an attribute in the list doesn't apply the layer when forwardpassing the input.

Usually the way to do this properly is to define new class which inherits from the model you want to implement - class myvgg16(models.vgg16) or more generally class myvgg(nn.Module). You can find further explanation in the following link

If it fails, try to unsqueeze(1) your targets size (i.e. the lables variable). This is less likly to be the reason for the error but worth a try.

EDIT

Give another try of converting your target tensor to one hot vectors. And change the tensor type to Float as the BCELoss receives floats.


Instruction: How to fix the dimension error in the loss function/softmax?

Input:
I am implementing a logistic regression in PyTorch for XOR (I don't expect it to work well it's just a demonstration). For some reason I am getting an error 'IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)'. It is not clear to me where this originates. The error points to log_softmax during training.

import torch.nn as nn
import torch.nn.functional as F

class LogisticRegression(nn.Module):
  # input_size: Dimensionality of input feature vector.
  # num_classes: The number of classes in the classification problem.
  def __init__(self, input_size, num_classes):
    # Always call the superclass (nn.Module) constructor first!
    super(LogisticRegression, self).__init__()
    # Set up the linear transform
    self.linear = nn.Linear(input_size, num_classes)

  # Forward's sole argument is the input.
  # input is of shape (batch_size, input_size)
  def forward(self, x):
    # Apply the linear transform.
    # out is of shape (batch_size, num_classes)
    out = self.linear(x)
    # Softmax the out tensor to get a log-probability distribution
    # over classes for each example.
    out_distribution = F.softmax(out, dim=-1)
    return out_distribution

# Binary classifiation
num_outputs = 1
num_input_features = 2

# Create the logistic regression model
logreg_clf = LogisticRegression(num_input_features, num_outputs)

print(logreg_clf)

lr_rate = 0.001

X = torch.Tensor([[0,0],[0,1], [1,0], [1,1]])
Y = torch.Tensor([0,1,1,0]).view(-1,1) #view is similar to numpy.reshape()

# Run the forward pass of the logistic regression model
sample_output = logreg_clf(X) #completely random at the moment
print(X)

loss_function = nn.CrossEntropyLoss() # computes softmax and then the cross entropy
optimizer = torch.optim.SGD(logreg_clf.parameters(), lr=lr_rate)

from torch.autograd import Variable
#training loop:

epochs = 201 #how many times we go through the training set
steps = X.size(0) #steps = 4; we have 4 training examples

for i in range(epochs):
    for j in range(steps):
        #sample from the training set:
        data_point = np.random.randint(X.size(0))
        x_var = Variable(X[data_point], requires_grad=False)
        y_var = Variable(Y[data_point], requires_grad=False)

        optimizer.zero_grad() # zero the gradient buffers
        y_hat = logreg_clf(x_var) #get the output from the model
        loss = loss_function.forward(y_hat, y_var) #calculate the loss
        loss.backward() #backprop
        optimizer.step() #does the update

    if i % 500 == 0:
        print ("Epoch: {0}, Loss: {1}, ".format(i, loss.data.numpy()))

Output:
First of all, you are doing a binary classification task. So the number of output features should be 2; i.e., num_outputs = 1.

Second, as it's been declared in nn.CrossEntropyLoss() documentation, the .forward method accepts two tensors as below:

Input: (N, C) where C is the number of classes (in your case it is 2).
Target: (N)

N in the example above is the number of training examples that you pass in to the loss function; for simplicity, you can set it to one (i.e., doing a forward pass for each instance and update gradients thereafter).

Note: Also, you don't need to use .Softmax() before nn.CrossEntropyLoss() module as this class has nn.LogSoftmax included in itself.

I modified your code as below, this is a working example of your snippet:

import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import torch

class LogisticRegression(nn.Module):
  # input_size: Dimensionality of input feature vector.
  # num_classes: The number of classes in the classification problem.
  def __init__(self, input_size, num_classes):
    # Always call the superclass (nn.Module) constructor first!
    super(LogisticRegression, self).__init__()
    # Set up the linear transform
    self.linear = nn.Linear(input_size, num_classes)

  # Forward's sole argument is the input.
  # input is of shape (batch_size, input_size)
  def forward(self, x):
    # Apply the linear transform.
    # out is of shape (batch_size, num_classes)
    out = self.linear(x)
    # Softmax the out tensor to get a log-probability distribution
    # over classes for each example.
    return out

# Binary classifiation
num_outputs = 2
num_input_features = 2

# Create the logistic regression model
logreg_clf = LogisticRegression(num_input_features, num_outputs)

print(logreg_clf)

lr_rate = 0.001

X = torch.Tensor([[0,0],[0,1], [1,0], [1,1]])
Y = torch.Tensor([0,1,1,0]).view(-1,1) #view is similar to numpy.reshape()

# Run the forward pass of the logistic regression model
sample_output = logreg_clf(X) #completely random at the moment
print(X)

loss_function = nn.CrossEntropyLoss() # computes softmax and then the cross entropy
optimizer = torch.optim.SGD(logreg_clf.parameters(), lr=lr_rate)

from torch.autograd import Variable
#training loop:

epochs = 201 #how many times we go through the training set
steps = X.size(0) #steps = 4; we have 4 training examples

for i in range(epochs):
    for j in range(steps):
        #sample from the training set:
        data_point = np.random.randint(X.size(0))
        x_var = Variable(X[data_point], requires_grad=False).unsqueeze(0)
        y_var = Variable(Y[data_point], requires_grad=False).long()

        optimizer.zero_grad() # zero the gradient buffers
        y_hat = logreg_clf(x_var) #get the output from the model
        loss = loss_function(y_hat, y_var) #calculate the loss
        loss.backward() #backprop
        optimizer.step() #does the update

    if i % 500 == 0:
        print ("Epoch: {0}, Loss: {1}, ".format(i, loss.data.numpy()))

Update

To get the predicted class labels which is either 0 or 1:

pred = np.argmax(y_hat.detach().numpy, axis=0)

As for the .detach() function, numpy expects the tensor/array to get detached from the computation graph; i.e., the tensor should not have require_grad=True and detach method would do the trick for you.