Instruction: What is happening with torch.Tensor.add_?

Input:
I'm looking at this implementation of SGD for PyTorch: https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD

And I see some strange calculations which I don't understand.

For instance, take a look at p.data.add_(-group['lr'], d_p). It makes sense to think that there is a multiplication of the two parameters, right? (It's how SGD works, -lr * grads)

But the documentation of the function doesn't say anything about this.

And what is more confusing, although this SGD code actually works (I tested by copying the code and calling prints below the add_), I can't simply use add_ with two arguments as it does:

#this returns an error about using too many arguments
import torch

a = torch.tensor([1,2,3])
b = torch.tensor([6,10,15])
c = torch.tensor([100,100,100])
a.add_(b, c)
print(a)

What's going on here? What am I missing?

Output:
This works for scalars:

a = t.tensor(1)
b = t.tensor(2)
c = t.tensor(3)
a.add_(b, c)
print(a)

  tensor(7)

Or a can be a tensor:

a = t.tensor([[1,1],[1,1]])
b = t.tensor(2)
c = t.tensor(3)
a.add_(b, c)
print(a)

  tensor([[7, 7],
          [7, 7]])

Output is 7, because: (Tensor other, Number alpha)


Instruction: Error when training CNN: "RuntimeError: The size of tensor a (10) must match the size of tensor b (64) at non-singleton dimension 1"

Input:
I'm new to Pytorch and I'm trying to implemente a simple CNN to recognize MNIST images.

I'm training the network using MSE Loss as loss function and SGD as optimizer. When I get to the training it gives me the following

  warning: " UserWarning: Using a target size (torch.Size([64])) that is different to the input size (torch.Size([64, 10])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size."

And then I get the following

error: "RuntimeError: The size of tensor a (10) must match the size of tensor b
       (64) at non-singleton dimension 1".

I've tried to solve it using some solutions I've found in other questions but nothing seems to work. Here's the code of how I load the dataset:

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,),(0.5,))])

trainset = torchvision.datasets.MNIST(root='./data', train = True, transform = transform, download = True)
trainloader = torch.utils.data.DataLoader(trainset, batch_size = 64, shuffle = True)

testset = torchvision.datasets.MNIST(root='./data', train = False, transform = transform, download = True)
testloader = torch.utils.data.DataLoader(testset, batch_size = 64, shuffle = False)

The code to define my network:

    class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #Convolutional layers
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 12, 5)
        #Fully connected layers
        self.fc1 = nn.Linear(12*4*4, 120)
        self.fc2 = nn.Linear(120, 60)
        self.out = nn.Linear(60,10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2,2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2,2))
        x = x.reshape(-1, 12*4*4)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.out(x)
        return x

And this is the training:

net = Net()
print(net)

criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.001)
epochs = 3

for epoch in range(epochs):
    running_loss = 0;
    for images, labels in trainloader:
        optimizer.zero_grad()
        output = net(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")

print('Finished training')

Thank you!

Output:
The loss you're using (nn.MSELoss) is incorrect for this problem. You should use nn.CrossEntropyLoss.
Mean Squared Loss measures the mean squared error between input x and target y. Here the input and target naturally should be of the same shape.
Cross Entropy Loss computes the probability over the classes for each image. The output would be a matrix N x C and target would be a vector of size N. (N = batch size, C = number of classes)
Since your aim is to classify the image, this is what you'll want to use.
In your case, your network output will be a matrix of size 64 x 10 and target is a vector of size 64. Each row of the output matrix (after applying the softmax function) indicates the probability of that class after which the Cross entropy loss is computed. Pytorch's nn.CrossEntropyLoss combines both the softmax operation with the loss computation.
You can refer the documentation here for more info on how Pytorch computes losses.


Instruction: Pytorch tutorial loss is not decreasing as expected

Input:
I am reimplementing the pytorch tutorial of the Pytorch cifar10 tutorial

But I want to use a different model.
I don't want to use fully connected (in pytorch linear) layers and I want to add Batch Normalization.

My model looks like this:

import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
  def __init__(self):
      super(Net, self).__init__()
      self.pool = nn.MaxPool2d(2,2)
      self.conv1 = nn.Conv2d(in_channels=3,out_channels=16,kernel_size=3, padding=1, padding_mode='zeros')
      self.conv1_bn = nn.BatchNorm2d(16)
      self.conv2 = nn.Conv2d(in_channels=16,out_channels=32,kernel_size=3, padding=1, padding_mode='zeros')
      self.conv2_bn = nn.BatchNorm2d(32)
      self.conv3 = nn.Conv2d(in_channels=32,out_channels=64,kernel_size=3, padding=1, padding_mode='zeros')
      self.conv3_bn = nn.BatchNorm2d(64)
      self.conv4 = nn.Conv2d(64,64,3, padding=1, padding_mode='zeros')
      self.conv4_bn = nn.BatchNorm2d(64)
      self.conv5 = nn.Conv2d(64,10,2,padding=0)
  def forward(self, x): # x has shape (4,32,32,3)
      x = self.pool(F.relu(self.conv1_bn(self.conv1(x)))) # feature map resolution is now 16*16
      x = self.pool(F.relu(self.conv2_bn(self.conv2(x)))) # resolution now 8*8
      x = self.pool(F.relu(self.conv3_bn(self.conv3(x)))) #resolution now 4*4
      x = self.pool(F.relu(self.conv4_bn(self.conv4(x)))) # now 2*2
      x = F.relu(self.conv5(x)) # The output shape is (batchsize, 1,1,10)

      return x

Batchsize is 4 and image resolution is 32*32 so inputsize is 4,32,32,3
The convolution layers don't reduce the resolution size of the feature maps because of the padding. The resolution is halved with the maxpool layers. Conv5 gets an input with shape 4,2,2,64.
Now I use filtersize 2 and no padding to get a resolution of 1*1.
I have 10 classes so I use 10 filters. Each of the last filters should predict it's corresponding class.
The shape of the output is now (4,1,1,10).
But when I try to train this model the loss doesn't decrease. The amount of parameters of the tutorial model and my net are about the same at ~62k.

Here is the rest of the code. This is identical to the code in the tutorial but I have to reshape the output so it fits. (output in the tutorial was (4,10) and mine is 4,1,1,10)

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

net = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net.to(device)
for epoch in range(2):  # loop over the dataset multiple times

  running_loss = 0.0
  for i, data in enumerate(trainloader, 0):
      # get the inputs; data is a list of [inputs, labels]
      inputs, labels = data[0].to(device), data[1].to(device)

      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = net(inputs) # I get the values as 4,1,1,10
      outputs_reshaped = outputs.reshape(4,10)
      loss = criterion(outputs_reshaped, labels)
      loss.backward()
      optimizer.step()
      running_loss +=loss.item()
      if i % 2000 == 1999:    # print every 2000 mini-batches
          print('[%d, %5d] loss: %.3f' %
                (epoch + 1, i + 1, running_loss / 2000))
          running_loss = 0.0

My loss looks like this.

[1,  2000] loss: 2.348
[1,  2000] loss: 2.477
[1,  4000] loss: 2.482
[1,  6000] loss: 2.468
[1,  8000] loss: 2.471
[1, 10000] loss: 2.482
[1, 12000] loss: 2.485
[2,  2000] loss: 2.486
[2,  4000] loss: 2.470
[2,  6000] loss: 2.479
[2,  8000] loss: 2.481
[2, 10000] loss: 2.474
[2, 12000] loss: 2.470

My model doesn't seem to learn anything. Anyone an idea why this might happen?

Output:
Your learning rate and momentum combination is too large for such a small batch size, try something like these:

optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.0)
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

Update: I just realized another problem is you are using a relu activation at the end of the network. If you look at the documentation of CrossEntropyLoss, there is an advice:

  The input is expected to contain raw, unnormalized scores for each
  class.

Try training your network by removing last relu from conv5 and keeping lr=0.01 and momentum=0.9. Relu before cross entropy loss throws away information about class scores.


Instruction: How do I load multiple grayscale images as a single tensor in pytorch?

Input:
I'm currently trying to use a stack a set of images as a single entity for each label to train a CNN on using cross-validation. Given a dataset of 224x224x1 grayscale images sorted by:

Root/
    Class0/image0_view0.png
    Class0/image0_view1.png
    Class0/image0_view2.png
    ...
    Class1/image0_view0.png
    Class1/image0_view1.png
    Class1/image0_view2.png

How would I go about flowing 3 images (view 0, 1, and 2) as a single tensor with dimensions 224x224x3 (3 grayscale images)? In other words, how would I create a dataset of image stacks in pytorch using ImageFolder/DatasetFolder and DataLoader? Would I have to re-organize my folders and classes, or would it be easier to make the stacks when I make the splits for cross-validation?

Thank you for your time and help! Let me know if I can provide any more info.

Output:
I had a very similar task. I needed to load a random sequence of 3 images as an element of a batch for training the network not on separate images but on seq of images. For batch size 8, I have 8 x 3 = 24 images. This seems to be very similar to different views in your case. I used imread_collection functionality from skimage.io. I added such a getitem to the Dataset class:
def __getitem__(self, idx):
    idx_q = int(torch.randint(0 + self.boundary, self.length - self.boundary, (1,)))

    q = imread_collection([self.image_paths[idx_q-1], self.image_paths[idx_q], self.image_paths[idx_q+1]], conserve_memory=True)

    if self.transform:
        q = torch.stack([self.transform(img) for img in q])

    return q, p, n

Here I generate a random index of an image and then load three consecutive images using imread_collection and self.image_paths, which is the list with paths to all images. Then I do transform of each image and stack them. In your case, you should think about using the right indexes, maybe by applying a sliding window on the length of self.image_paths.
A bit more info could be found on the torch forum. I also tried to ask and find a more elegant solution, but couldn't and successfully trained the model with such an approach.


Instruction: fastai - Multiclass metric for Image Segmentation

Input:
I’m currently exploring how to apply Dice metric to a multiclass segmentation problem with fastai. I checked the concepts and discovered that Dice is really similar to the F1Score. Following this, I have two questions regarding their implementation in fastai.metrics :

How different are the dice() and fbeta(beta=1) outputs ?
Is the MultiLabelFbeta class okay for multilabel image segmentation use cases ?
If not, is there a metric already existing that could help me ?

Thank you very much and have a good day !

Output:
The Dice metric should normally be equal to FBeta(beta=1). Depending on the framework, there may be slight differences in the implementation. However, since these are by nature very similar, they can be used interchangeably as metrics for your problem.

MultiLabelFBeta can be used if you have multiple overlapping masks. That is, if your segmentation labels are not mutually exclusive.

For example, a dog and a cat have pixels that are mutually exclusive(i.e. a pixel belonging to a cat will never belong to a dog and vice-versa). However, if you have the class 'T-shirt' and 'Human' then it is evident that you have overlapping: people wear t-shirts, so a pixel that belongs to T-Shirt may very well belong to a human being.

Pay attention to the nomenclature! MultiLabel is different from MultiClass. In case of the latter, the labels are mutually exclusive; in case of the former, they are not (T-Shirt + Human example).

If you have a multi-class segmentation problem, then Dice/FBeta are relevant metrics. If you have a multi-label segmentation problem, then MultiLabelFbeta is a good metric.


Instruction: My Google Colab keeps crashing because of something in this code. Not sure what it is

Input:
My google colab keeps crashing at train, even though RAM and disk are plenty. The problem is somewhere in this code, I think, but I don't know what it is. I am doing LSTM. I'd appreciate any help. I am using PyTorch.

This code is followed by generate function, encoder decoder class, etc. It is when I get to train that it crashes (for "unknown reason")

class LSTMLM(torch.nn.Module):
  def __init__(self,
              vocab_size,
              embedding_size,
              hidden_size,
              num_layers=1,
              dropout=0.1):
    super().__init__()
    self.vocab_size = vocab_size
    self.embedding_size = embedding_size
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.dropout = torch.nn.Dropout(dropout)

    self.embedding = torch.nn.Embedding(vocab_size, embedding_size)

    self.lstm = torch.nn.LSTM(input_size=hidden_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)

    self.logistl = torch.nn.Linear(hidden_size, vocab_size)

    pass

  def forward(self, x, init_hidden_state=None):
    assert x.shape[0] == 1

    emb = self.embedding(x)

    emb = self.dropout(emb)

    if init_hidden_state is None:
      h0 = torch.zeros(self.num_layers, 1, self.hidden_size)
      c0 = torch.zeros(self.num_layers, 1, self.hidden_size)
    else:
      h0, c0 = init_hidden_state
    output, (hn, hc) = self.lstm(emb, (h0, c0))

    hidden_states = output
    final_hidden_state = hn
    final_cell_state = cn
    final_state = [final_hidden__state, final_cell_state]

    hidden_states = self.dropout(hidden_states)

    output_dist = self.logistl(hidden_states)

    return output_dist, hidden_states, final_state

Output:
The solution was to add cuda() after defining each tensor.

E.g.

a = a.cuda()

Note that you must assign it instead of doing just a.cuda()


Instruction: Output of a neural network

Input:
I test my network with the mnist dataset. Therefore the output of the model has shape 10.

How do I reshape the output?
For example, if the output is label 3, is then the output [0 0 0 1 0 0 0 0 0 0], or [0 0 0 3 0 0 0 0 0 0] or completely different?

The thing is I don't want to use the dataloader. I use this method:

from mlxtend.data import loadlocal_mnist
X, y = loadlocal_mnist(
            images_path='/home/wai043/data/mnist/train-images-idx3-ubyte',
            labels_path='/home/wai043/data/mnist/train-labels-idx1-ubyte')

Output:
The output has to be [0, 0, 0, 1, 0, 0, 0, 0, 0, 0] if the label is 3. The y parameter you get from loadlocal_mnist has the direct label, so you need to "one-hot encode" y before your training.

You can use the following code to do the encoding

from mlxtend.preprocessing import one_hot
from mlxtend.data import loadlocal_mnist

X, y = loadlocal_mnist(images_path='/home/wai043/data/mnist/train-images-idx3-ubyte',
                       labels_path='/home/wai043/data/mnist/train-labels-idx1-ubyte')
y = one_hot(y)


Instruction: What is an efficient way to read and store input data from multiple files in python?

Input:
I want to train a classifier for 8000 data samples in python (using sklearn or pytorch libraries). But, each data is a times-series stored in a .wav file.

Do I have to first read all 8000 files in a loop and store them in a big file, and read that big file each time I want to train a classifier? or there is a more efficient way for such cases?

Output:
No you don't need to have one big file.
At some point you may lack the GPU memory (memory overall).
What you need is lazy loading.

In PyTorch you have the Dataset and DataLoader classes.

Dataset just need to implement two methods.

class Dataset(object):

    def __getitem__(self, index):
        raise NotImplementedError

    def __len__(self):
        raise NotImplementedError

DataLoader class is there to provide you the single batch of data. If batch size is 64  (bs=64) this means 64 .wav files from once. So you will instruct the DataLoader to load bs of data from once. This is lazy loading.


Instruction: How to Multi-Head learning

Input:
I have about 5 models that work pretty well trained individually but I want to fuse them together in order to have one big model.
I'm looking into it because one big model is more easy to update (in production) than many small model
this is an image of what I want to achieve.

my question are, is it ok to do it like this ?
having one dataset per head model, how am I supposed to train the whole model ?

Output:
  my question are, is it ok to do it like this

Sure you can do that. This approach is called multi-task learning. Depending on your datasets and what you are trying to do, it will maybe even increase the performance. Microsoft used a multi-task model to achieve some good results for the NLP Glue benchmark, but they also noted that you can increase the performance further by finetuning the joint model for each individual task.

  having one dataset per head model, how am I supposed to train the whole model?

All you need is pytorch ModuleList:

#please note this is just pseudocode and I'm not well versed with computer vision
#therefore you need to check if resnet50 import is correct and look
#for the imports of the task specific stuff
from torch import nn
from torchvision.models import resnet50

class MultiTaskModel(nn.Module):
    def __init__(self):
        #shared part
        self.resnet50 = resnet50()

        #task specific stuff
        self.tasks = nn.ModuleList()
        self.tasks.add_module('depth', Depth())
        self.tasks.add_module('denseflow', Denseflow())
        #...

    def forward(self, tasktag, ...):
        #shared part
        resnet_output = self.resnet50(...)

        #task specific parts
        if tasktag == 'depth':
            return self.tasks.depth(resnet_output)
        elif tasktag == 'denseflow':
            return self.tasks.denseflow(resnet_output)
        #...


Instruction: How to normalize PIL image between -1 and 1 in pytorch for transforms.Compose?

Input:
So I have been trying to find a way to normalize some PIL image pixel values between -1 and 1. I searched through documentation and didn't find solution. Only normalization in documentation is transforms.Normalize which normalizes with mean and std. So I am stuck on how to do it. This is my code:

train_transform = transforms.Compose([
        transforms.RandomHorizontalFlip(p=0.5),
        transforms.Resize(40),
        transforms.RandomCrop(32),
        # Normalize(-1, 1)  # Something like that
])

How can I normalize tensor between 2 numbers? For example:

[[1,2,3]
[3,2,1]]

Between -1 and 1

[[-1, 0, 1],
[1, 0, -1]]

Output:
Try:

transform = transforms.Compose([
    transforms.RandomHorizontalFlip(p=0.5)
    transforms.Resize(40),
    transforms.RandomCrop(32),
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))])

ToTensor will scale image pixels from [0,255] to [0,1] values
Normalize will first subtract the mean so [0,1] values will transform to range [-0.5,0.5], and then will divide with std and [-0.5,0.5] will reach [-1,1] range


Instruction: What does group.setdefault() do in this context?

Input:
I looked at the documentation [https://docs.python.org/2/library/stdtypes.html#dict.setdefault]

  setdefault(key[, default]):

  If key is in the dictionary, return its value.

  If not, insert key with a value of default and return default. default defaults to None

However, seeing as the output of group.setdefault() isn't being saved in a variable. I find this odd.
To be more specific, this code is included in the definition of a class of a scheduler. I am using pytorch and as I'm new, I'm having difficulties understanding the code. So any python/pytorch ninja that can figure this out?

Here's a simplified version of the class:

     class BaseLRScheduler(_LRScheduler):
         def __init__(self, optimizer, last_epoch=-1):

             self.optimizer = optimizer
             if last_epoch == -1:
                for group in optimizer.param_groups:
                    group.setdefault('initial_lr', group['lr'])

             self.base_lrs = list(
                  map(lambda group: group['initial_lr'], optimizer.param_groups))
             self.step(last_epoch + 1)
             self.last_epoch = last_epoch

This is the problematic part of code for me:

     if last_epoch == -1:
         for group in optimizer.param_groups:
              group.setdefault('initial_lr', group['lr'])

Output:
The fact that a function returns a value does not make it mandatory to use that value.

In this case,
if 'initial_lr' is in the dictionary, nothing happens, but,
If 'initial_lr' is not in the dictionary, insert it with the value of group['lr'].

For example:

d = {'a': 1}
d.setdefault('a', 2)
print(d)
d.setdefault('b', 3)
print(d)

Outputs

{'a': 1}
{'a': 1, 'b': 3}


Instruction: How to parallelize CNN evaluation of images with PyTorch

Input:
I have a fairly long script that uses a CNN to classify people from drone footage as either human or non-human. The general flow is as such: (1) Create a video object and extract "captures" from it based on intervals of seconds specified. (2) Instantiate the Model class, loading a PyTorch CNN. (3) For each capture, break into much smaller (and overlapping) images which the CNN will classify as human or non-human. (4) Create a list of coordinates, which satisfy previous step. (5) Draw red squares surrounding coordinates, and save marked up image. (6) Repeat process for every capture.

There's a bottleneck at the image level. Crops are being produced and evaluated by the CNN sequentially. I would love to parallelize this process but it goes beyond my current knowledge/experience level.

Any recommendations? The script is below for reference.

import torch, torchvision
from torchvision import datasets, models, transforms
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
import time
from torchsummary import summary

import numpy as np
import matplotlib.pyplot as plt
import os

from PIL import Image
import shutil
from PIL import Image, ImageDraw
import random
import cv2

class Model():

    def __init__(self,model):

        self.idx_to_class = {1:'No human',0:'Human'}
        self.image_transforms = {
            'test': transforms.Compose([
                transforms.Resize(size=256),
                transforms.CenterCrop(size=224),
                transforms.ToTensor(),
                transforms.Normalize([0.485, 0.456, 0.406],
                                     [0.229, 0.224, 0.225])
            ])
        }
        self.model = torch.load(model)

    def predict(self, test_image_name):
        transform = self.image_transforms['test']
#         test_image = Image.open(test_image_name)
        test_image_tensor = transform(test_image_name)

        if torch.cuda.is_available():
            test_image_tensor = test_image_tensor.view(1, 3, 224, 224).cuda()
        else:
            test_image_tensor = test_image_tensor.view(1, 3, 224, 224)

        with torch.no_grad():
            self.model.eval()
            out = self.model(test_image_tensor)
            ps = torch.exp(out)
            topk, topclass = ps.topk(1, dim=1)
            return topclass.cpu().numpy()[0][0]

class Image_classifier():

    def __init__(self,image,folder,positive_location,model):
        self.name = image
        self.alias = image.split('.')[0]
        self.folder = folder
        self.src = Image.open(f"{self.folder}/{self.name}")
        self.width = self.src.size[0]
        self.square_size = int(self.width/25)
        self.max_down = int(self.src.height/self.square_size) * self.square_size - self.square_size
        self.max_right = int(self.src.width/self.square_size) * self.square_size - self.square_size
        self.offset = int(self.square_size/3)
        self.positive_location = positive_location
        self.model = model

    def window_coordinates(self):

        def right_pass(y):
            x_coords = [x for x in range(0,self.max_right,self.offset)]
            y_coords = [y for x in range(0,self.max_right,self.offset)]
            return [(x,y,x+self.square_size,y+self.square_size) for x,y in zip(x_coords,y_coords)]

        #v_pass = np.vectorize(right_pass)

        y_values = [y for y in range(0,self.max_down,self.offset)]
        coordinates = [right_pass(y) for y in y_values]
        self.coordinates = [item for sublist in coordinates for item in sublist]

    def predict_coord(self,coord):
        sample = self.src.crop(coord)
        return self.model.predict(sample)

    def parse_coordinates(self):

        new_coords = [coord if self.predict_coord(coord) == 0 else 0 for coord in self.coordinates]
        while 0 in new_coords:
            new_coords.remove(0)
        self.coordinates = new_coords

    def select_squares(self):
        self.window_coordinates()
        self.parse_coordinates()
        self.drawable = ImageDraw.Draw(self.src)
        for coord in self.coordinates:
            self.drawable.rectangle(list(coord), fill = None, outline = 'red')
        self.src.save(f"{self.positive_location}/{self.alias}.jpg")

class Video_classifier():
    def __init__(self,video,root,seconds):
        self.video = video
        self.alias = self.video.split('.')[0]
        self.root = root
        self.seconds = seconds
        self.model = Model(model='/home/team4/output/_model_142.pt')

        self.folder = f"{self.root}/{self.alias}"
        if os.path.exists(self.folder):
            shutil.rmtree(self.folder)
        os.mkdir(self.folder)

        self.positive_location = f"{self.root}/{self.alias}/positives"
        if os.path.exists(self.positive_location):
            shutil.rmtree(self.positive_location)
        os.mkdir(self.positive_location)

    def get_frames(self):
        import cv2
        cam = cv2.VideoCapture(f"{self.root}/{self.video}")

        (major_ver, minor_ver, subminor_ver) = (cv2.__version__).split('.')
        if int(major_ver)  < 3 :
            fps = round(video.get(cv2.cv.CV_CAP_PROP_FPS))
        else:
            fps = round(cam.get(cv2.CAP_PROP_FPS))

        current_frame = 0
        while(True):
            ret,frame = cam.read()
            if ret:
                if current_frame % (self.seconds*fps) == 0:
                    f_name = f"{self.alias}_{current_frame}.jpg"
                    cv2.imwrite(f_name, frame)
                    shutil.move(f_name, self.folder)

                current_frame += 1
            else:
                break

        cam.release()
        cv2.destroyAllWindows()

    def read_dir(self):
        self.files = [f for f in os.listdir(self.folder) if os.path.isfile(os.path.join(self.folder, f))]

    def classify_frames(self):
        self.get_frames()
        self.read_dir()
        for file in self.files:
            image = Image_classifier(image=file, folder=self.folder, positive_location=self.positive_location,
                                     model=self.model)
            image.select_squares()

test = Video_classifier(video='refugee_test.mp4',root='/home/team4/Untitled Folder 1', seconds=10)
test.classify_frames()

I apologize for the lack of comments, doc-strings, etc. It's the in the works.

Output:
So you are gonna want to learn how to use the library multiprocessing. There are no doubt many ways to go about this as that library is extensive. Here is how I would go about it assuming you need to keep track of the different processes to put the image back together correctly. This is more complicated than just creating a process pool but allows you to keep tack of all processes.

from multiprocessing import Process, Pipe

#keeps track of connections
conList = []
for i in range(numberOfProcessesNeeded):
   #creates connection
   recv, send = Pipe()

   #gets process ready with function to complete and arguments
   pid = Process(target=imageProcessingFunction, args=(i,command,send))

   #starts process
   pid.start()

   #keeps track of process
   conList.append([i,recv])

Now one of the arguments to that function must be 'send' from the pipe creation so that when the process ends it can send information back.

In the Image processing function that you want to use 'send.send(DataToReturn)' instead of 'return(DataToReturn)'.

In the main function to collect the data do this.

data = []
for i in conList:
    #this will wait until each process has finished and then collect the data
    out = i[1].recv()
    data.append(out)

After this the array data will have all of your images in the order you called them to be processed.

Hopefully that wasn't to confusing but this is a tricky subject. More info can be found here:
https://docs.python.org/3.4/library/multiprocessing.html


Instruction: PyTorch CNN never converges (implementation issue suspected)

Input:
I am having trouble getting this network to work as desired. I have tried so many iterations of this model and yet cannot get a reasonable error (it never fits, can’t even get it to overfit).

Where have I gone wrong? Any help would be greatly appreciated

For reference, there are 12 input ‘images’ (they’re actually water surface elevation at 9 stations in an estuary) of shape 49,9 and 12 labels of shape 1,9.

Full examples with data can be found at https://gitlab.com/jb4earth/effonn/

  net = []
  class Net(torch.nn.Module):
      def __init__(self, kernel_size):
          super(Net, self).__init__()
          mid_size = (49*49*9)
          self.predict = torch.nn.Sequential(
              nn.Conv2d(
                          in_channels=1,
                          out_channels=mid_size,
                          kernel_size=kernel_size,
                          stride=1,
                          padding=(0, 0)
                      ),
              nn.ReLU(),
              nn.MaxPool2d(1),
              nn.ReLU(),
              nn.Conv2d(
                          in_channels=mid_size,
                          out_channels=1,
                          kernel_size=kernel_size,
                          stride=1,
                          padding=(0, 0)
                      ),
              nn.ReLU()
          )

      def forward(self, x):
          x = self.predict(x)
          return x

  def train_network(x,y,optimizer,loss_func):
      prediction = net(x)
      loss = loss_func(prediction, y.squeeze())
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      return prediction, loss

  net = Net((1,1))
  optimizer = torch.optim.Adam(net.parameters(), lr=0.01)
  loss_func = torch.nn.MSELoss()
  cnt = 0
  t = True
  while t == True:
      # get_xy in place of DataLoader
      (x,y) = get_xy(input_data,output_data,cnt)
      # x.shape is 1,1,49,9
      # y.shape is 1,1,1,9

      # train and predict
      (prediction,loss) = train_network(x,y,optimizer,loss_func)

      # prediction shape different than desired so averaging all results
      prediction_ = torch.mean(prediction)

      # only 12 IO's so loop through
      cnt += 1
      if cnt > 11:
          cnt = 0

Output:
take a look here, this looks suspicious. you are calculating the loss and then making the gradients zeros. calling zero grad should be called before calculating the loss. So you need to switch the optimizer.zero_grad() to the top and I assume it will work. I couldn't reproduce your example that's why I'm guessing this is your Error.

  loss = loss_func(prediction, y.squeeze())
  optimizer.zero_grad()   # switch this to the top
  loss.backward()
  optimizer.step()


Instruction: set `torch.backends.cudnn.benchmark = True` or not?

Input: I am using pytorch and I wonder if I should use torch.backends.cudnn.benchmark = True. I find on google that I should use it when computation graph does not change. What is computation graph in pytorch?

Output:
If your model does not change and your input sizes remain the same - then you may benefit from setting torch.backends.cudnn.benchmark = True.
However, if your model changes: for instance, if you have layers that are only "activated" when certain conditions are met, or you have layers inside a loop that can be iterated a different number of times, then setting torch.backends.cudnn.benchmark = True might stall your execution.


Instruction: 2.3 ratio between Pytorch BCEloss and my own "log" calculations

Input:
I'm scripting a toy model to both practice PyTorch and GAN models, and I'm making sure I understand each step as much as possible.
That leaded me to checking my understanding of the BCEloss function, and apparently I understand it... with a ratio of 2.3.

To check the results, I write the intermediate values for Excel:

tmp1 = y_pred.tolist()  # predicted values in list (to copy/paste on Excel)
tmploss = nn.BCELoss(reduction='none')  # redefining a loss giving the whole BCEloss tensor
tmp2 = tmploss(y_pred, y_real).tolist()  # BCEloss values in list (to copy/paste Exel)

Then I copy tmp1 on Excel and calculate: -log(x) for each values, which is the BCEloss formula for y_target = y_real = 1.

Then I compare the resulting values with the values of tmp2: these values are 2.3x higher than "mine".

(Sorry, I couldn't figure out how to format tables on this site...)

Can you please tell me what is happening? I feel a PEBCAK coming :-)

Output:
This is because in Excel the Log function calculates the logarithm to the base 10.
The standard definition of binary cross entropy uses a log function to the base e.
The ratio you're seeing is just log(10)=2.302585


Instruction: How to accelerate the code below? Implementing maxpool without center element

Input:
I know maxpool and I am using it in pytorch. Maxpool with dilated parameter is as belows:

Now I want a special form of maxpool, doing maxpool with out the central element. That is the kernel size is 3X3 but the central element should be deleted. Thus the result should come from the rest 8 elements.
Now I am using a for loop, how to accelerate this using numpy or pytorch or anything else?

import numpy as np
from timeit import default_timer as timer

def MaxPool_special(kh, kw, arr):
    """
    to do maxpool without central element
    :param kh:  should always be 3
    :param kw:  should always be 3
    :param arr:  the input array
    :return: arr_res: output array
    """
    h, w = arr.shape[:2]

    arr_res = np.array([[maxpool_ij(i, j, arr, kh, kw) for j in range(w)] for i in range(h)])

    return arr_res

def maxpool_ij(i, j, arr, dh, dw):
    """
    find the maximum value around point(i,j) with dilated parameter
    """
    Mmax = None
    imin, imax = i - dh, i + dh
    jmin, jmax = j - dw, j + dw
    if imin >= 0 and imax < h and jmin >= 0 and jmax < w:
        Mmax = np.max(
            arr[[imin, imin, imin, i, i, imax, imax, imax], [jmin, j, jmax, jmin, jmax, jmin, j, jmax]])
    elif imin < 0 and jmin < 0:
        Mmax = np.max(arr[[i, imax, imax], [jmax, j, jmax]])
    elif imin < 0 and jmax >= w:
        Mmax = np.max(arr[[i, imax, imax], [jmin, jmin, j]])
    elif imax >= h and jmin < 0:
        Mmax = np.max(arr[[imin, imin, i], [j, jmax, jmax]])
    elif imax >= h and jmax >= w:
        Mmax = np.max(arr[[imin, imin, i], [jmin, j, jmin]])
    elif imin < 0:
        Mmax = np.max(arr[[i, i, imax, imax, imax], [jmin, jmax, jmin, j, jmax]])
    elif imax >= h:
        Mmax = np.max(arr[[imin, imin, imin, i, i], [jmin, j, jmax, jmin, jmax]])
    elif jmin < 0:
        Mmax = np.max(arr[[imin, imin, i, imax, imax], [j, jmax, jmax, j, jmax]])
    elif jmax >= w:
        Mmax = np.max(arr[[imin, imin, i, imax, imax], [jmin, j, jmin, jmin, j]])

    assert Mmax, f'Wrong logic above!{imin, imax, jmin, jmax, h, w}'

    return Mmax

#  generate input array
h, w = 400, 500
arr = np.random.randint(0, 256, h * w).reshape(h, w)

tic = timer()
grayPool = MaxPool_special(3, 3, arr)
toc = timer()
print(f'time cost for for-loops: {toc - tic}')

Please help me accelerating this code, thanks!

Output:
Using torch.nn.Unfold can implement Maxpool without central element. Examples are as below:

h, w = 7, 10
x = torch.arange(0,h*w,dtype=torch.float).reshape(1,1,h,w)

"""
x.shape: torch.Size([1, 1, 7, 10])
x:
tensor([[[[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.],
          [10., 11., 12., 13., 14., 15., 16., 17., 18., 19.],
          [20., 21., 22., 23., 24., 25., 26., 27., 28., 29.],
          [30., 31., 32., 33., 34., 35., 36., 37., 38., 39.],
          [40., 41., 42., 43., 44., 45., 46., 47., 48., 49.],
          [50., 51., 52., 53., 54., 55., 56., 57., 58., 59.],
          [60., 61., 62., 63., 64., 65., 66., 67., 68., 69.]]]])
"""

Unfold = torch.nn.Unfold(kernel_size=(3,5), stride=(2,2))
xUfd = Unfold(x)
"""
xUfd.shape: torch.Size([1, 15, 9])
xUfd:
tensor([[[ 0.,  2.,  4., 20., 22., 24., 40., 42., 44.],
         [ 1.,  3.,  5., 21., 23., 25., 41., 43., 45.],
         [ 2.,  4.,  6., 22., 24., 26., 42., 44., 46.],
         [ 3.,  5.,  7., 23., 25., 27., 43., 45., 47.],
         [ 4.,  6.,  8., 24., 26., 28., 44., 46., 48.],
         [10., 12., 14., 30., 32., 34., 50., 52., 54.],
         [11., 13., 15., 31., 33., 35., 51., 53., 55.],
         [12., 14., 16., 32., 34., 36., 52., 54., 56.],
         [13., 15., 17., 33., 35., 37., 53., 55., 57.],
         [14., 16., 18., 34., 36., 38., 54., 56., 58.],
         [20., 22., 24., 40., 42., 44., 60., 62., 64.],
         [21., 23., 25., 41., 43., 45., 61., 63., 65.],
         [22., 24., 26., 42., 44., 46., 62., 64., 66.],
         [23., 25., 27., 43., 45., 47., 63., 65., 67.],
         [24., 26., 28., 44., 46., 48., 64., 66., 68.]]])
"""

xUfd = xUfd[:,:,[0,1,2,3,5,6,7,8]]
xUfd = torch.max(xUfd, 2).values.reshape(3,5)
"""
xUfd.shape: torch.Size([3, 5])
xUfd:
tensor([[44., 45., 46., 47., 48.],
        [54., 55., 56., 57., 58.],
        [64., 65., 66., 67., 68.]])
"""


Instruction: Given groups=1, weight of size 16 16 3 3, expected input[16, 64, 222, 222] to have 16 channels, but got 64 channels instead?

Input:
I try to run the following program for image classification problems in Pytorch.
I am new to PyTorch and I am not getting what's wrong with the code. I tried reshaping the images but no help.
I am running this code with Cuda. I have around 750 classes and 10-20 images in one class. My dataset is a benchmark dataset and every image has a size of 60*160.

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.ConvLayer1 = nn.Sequential(
            nn.Conv2d(3, 64, 3), # inp (3, 512, 512) changes doing here original (3, 64, 3)
            nn.Conv2d(8, 16, 3), # original (8,16,3)
            nn.MaxPool2d(2),
            nn.ReLU() # op (16, 256, 256)
        )
        self.ConvLayer2 = nn.Sequential(
            nn.Conv2d(16, 32, 5), # inp (16, 256, 256)
            nn.Conv2d(32, 32, 3),
            nn.MaxPool2d(4),
            nn.ReLU() # op (32, 64, 64)
        )
        self.ConvLayer3 = nn.Sequential(
            nn.Conv2d(32, 64, 3), # inp (32, 64, 64) original (32,64,3)
            nn.Conv2d(64, 64, 5),
            nn.MaxPool2d(2),
            nn.ReLU() # op (64, 32, 32)
        )
        self.ConvLayer4 = nn.Sequential(
            nn.Conv2d(64, 128, 5), # inp (64, 32, 32)
            nn.Conv2d(128, 128, 3),
            nn.MaxPool2d(2),
            nn.ReLU() # op (128, 16, 16)
        )
        self.Lin1 = nn.Linear(15488, 15)
        self.Lin2 = nn.Linear(1500, 150)
        self.Lin3 = nn.Linear(150, 15)

    def forward(self, x):
        x = self.ConvLayer1(x)
        x = self.ConvLayer2(x)
        x = self.ConvLayer3(x)
        x = self.ConvLayer4(x)
        x = x.view(x.size(0), -1)
        x = self.Lin1(x)

        return F.log_softmax(x, dim = 1)
'''

'''
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.0001, momentum=0.5)
for epoch in tqdm(range(2)):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(dataloaders['train']):
        # get the inputs; data is a list of [inputs, labels]
        inputs, class_names = data
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, class_names)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
       # if i % 10 == 0:    # print every 10 mini-batches
        #    print('[%d, %5d] loss: %.3f' %
         #         (epoch + 1, i + 1, running_loss / 2000))
          #  running_loss = 0.0
    break
print('Finished Training')

Getting this error and I don't know where to make changes.
Given groups=1, the weight of size 16 16 3 3, expected input[16, 64, 222, 222] to have 16 channels, but got 64 channels instead.

Output: The number of output channels in a conv layer need to match the number of input channels in the next conv layer. Say you have nn.Conv(3, 64, 3) then the next conv layer needs to begin nn.Conv(64, .... Right now the issue is that you're trying to pass a 64 channel result into a conv layer which you've defined to expect an 8 channel input.


Instruction: Pytorch error on multi-GPU setup: cannot import name 'invoke_remote_python_udf' from 'torch.distributed'

Input:
I'm getting this below error when importing Pytorch.

"cannot import name 'invoke_remote_python_udf' from 'torch.distributed'"

Output:
I faced the similar problem. For me , the problem was occuring due to mismatch in cuda version. I am using conda environment.

In my system the nvcc --version showed the result as 10.0' , while I had mistakenly install the pytorch for cuda version10.1`

I removed the torch and torchvision and re-installed them using the following command:

conda install pytorch torchvision cuda100 -c pytorch.

Installing the package cuda100 solved the problem.


Instruction: Optimizing a neural network with a multi-task objective in Pytorch

Input:
In deep learning, you typically have an objective (say, image recognition), that you wish to optimize. In my field (natural language processing), though, we've seen a rise of multitask training. For instance, in next sentence prediction and sentence classification in a single system.

I understand how to build the forward pass, e.g. for a classification task (obj1) and a regression task (obj2)

class Net(nn.Module):
    def __init__():
        super().__init__()
        self.linear = Linear(300, 200)
        self.obj1 = Linear(200, 5)
        self.obj2 = Linear(200, 1)

    def forward(inputs):
        out = self.linear(inputs)
        out_obj1 = self.obj1(out)
        out_obj2 = self.obj2(out)
        return out_obj1, out_obj2

But the question then becomes, how does one optimize this. Do you call a backward pass over both losses separately? Or do you reduce them to a single loss (e.g. sum, average)? Is there an approach that is typically used for multi-task learning?

And to follow up on that, perhaps one could even argue that the parameters of the separate layers need different optimizers. In such case, the losses must be dealt with separately, I presume.

Output:
It is much simpler, you can optimize all variables at the same time without a problem. Just compute both losses with their respective criterions, add those in a single variable:

total_loss = loss_1 + loss_2

and calling .backward() on this total loss (still a Tensor), works perfectly fine for both. You could also weight the losses to give more importance to one rather than the other.

Check the PyTorch forums for more information.


Instruction: BertTokenizer - when encoding and decoding sequences extra spaces appear

Input:
When using Transformers from HuggingFace I am facing a problem with the encoding and decoding method.

I have a the following string:

test_string = 'text with percentage%'

Then I am running the following code:

import torch
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

test_string = 'text with percentage%'

# encode Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary.
input_ids = tokenizer.encode(test_string)
output = tokenizer.decode(input_ids)

And the output looks like this:

'text with percentage %'

With an extra space before the %. I have tried the extra arguments like clean_up_tokenization_spaces  but this is for something different.

How what should I use in the decoding and encoding to get exactly the same text before and after. This also happens for other special signs.

Output:
If you are trying to use BERT for token classification in order to find a span in your original string, then one workaround is to use BertTokenizerFast with the option return_offsets_mapping=True.
test_string = 'text with percentage%'

tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
tokens = tokenizer(test_string, return_offsets_mapping=True)
input_ids = tokens.data["input_ids"]

span_start_index, span_stop_index = some_model(input_ids)

Then once you get the token classification results, you can do something like
predicted_span = test_string[tokens.encodings[0].offsets[span_start_index][0]:tokens.encodings[0].offsets[span_stop_index][1]]


Instruction: what is the biggest bottleneck in maskrcnn_benchmark repo?

Input:
I am working on a repo that make use of the maskrcnn_benchmark repo. I have extensively, explored the bench-marking repo extensively for the cause of its slower performance on a cpu with respect to enter link description here.

In order to create a benchmark for the individual forward passes I have put a time counter for each part and it gives me the time required to calculate each component. I have had a tough time exactly pinpointing as to the slowest component of the entire architecture.I believe it to be BottleneckWithFixedBatchNorm class in the maskrcnn_benchmark/modeling/backbone/resnet.py file.

I will really appreciate any help in localisation of the biggest bottle neck in this architecture.

Output:
I have faced the same problem, the best possible solution for the same is to look inside the main code, go through the forward pass of each module and have a timer setup to log the time that is spent in the computations of each module. How we worked in it was to create an architecture where we create the time logger for each class, therefore every instance of the class will now be logging its time of execution, after through comparison, atleast in our case we have found that the reason for the delay was the depth of the Resnet module, (which given the computational cost of resnet is not a surprising factor at all, the only solution to the same is more palatalization so either ensure a bigger GPU for performing the task or reduce the depth of the Resnet network ).

I must inform that the maskrcnn_benchmark has been deprecated and an updated version of the same is available in the form of detectron2. Consider moving your code for significant speed improvements in the architecture.

BottleneckWithFixedBatchNorm is not the most expensive operation in the architecture and certainly not creating the bottleneck as all the operations instead of the name. The class isn't as computationally expensive and is computed in parallel even on a lower end CPU machine (at least in the inference stage).

An example of tracking better the performance of each module can be found with the code taken from the path : maskrcnn_benchmark/modeling/backbone/resnet.py

class ResNet(nn.Module):
    def __init__(self, cfg):
        super(ResNet, self).__init__()

        # If we want to use the cfg in forward(), then we should make a copy
        # of it and store it for later use:
        # self.cfg = cfg.clone()

        # Translate string names to implementations
        stem_module = _STEM_MODULES[cfg.MODEL.RESNETS.STEM_FUNC]
        stage_specs = _STAGE_SPECS[cfg.MODEL.BACKBONE.CONV_BODY]
        transformation_module = _TRANSFORMATION_MODULES[cfg.MODEL.RESNETS.TRANS_FUNC]

        # Construct the stem module
        self.stem = stem_module(cfg)

        # Constuct the specified ResNet stages
        num_groups = cfg.MODEL.RESNETS.NUM_GROUPS
        width_per_group = cfg.MODEL.RESNETS.WIDTH_PER_GROUP
        in_channels = cfg.MODEL.RESNETS.STEM_OUT_CHANNELS
        stage2_bottleneck_channels = num_groups * width_per_group
        stage2_out_channels = cfg.MODEL.RESNETS.RES2_OUT_CHANNELS
        self.stages = []
        self.return_features = {}
        for stage_spec in stage_specs:
            name = "layer" + str(stage_spec.index)
            stage2_relative_factor = 2 ** (stage_spec.index - 1)
            bottleneck_channels = stage2_bottleneck_channels * stage2_relative_factor
            out_channels = stage2_out_channels * stage2_relative_factor
            stage_with_dcn = cfg.MODEL.RESNETS.STAGE_WITH_DCN[stage_spec.index -1]
            module = _make_stage(
                transformation_module,
                in_channels,
                bottleneck_channels,
                out_channels,
                stage_spec.block_count,
                num_groups,
                cfg.MODEL.RESNETS.STRIDE_IN_1X1,
                first_stride=int(stage_spec.index > 1) + 1,
                dcn_config={
                    "stage_with_dcn": stage_with_dcn,
                    "with_modulated_dcn": cfg.MODEL.RESNETS.WITH_MODULATED_DCN,
                    "deformable_groups": cfg.MODEL.RESNETS.DEFORMABLE_GROUPS,
                }
            )
            in_channels = out_channels
            self.add_module(name, module)
            self.stages.append(name)
            self.return_features[name] = stage_spec.return_features

        # Optionally freeze (requires_grad=False) parts of the backbone
        self._freeze_backbone(cfg.MODEL.BACKBONE.FREEZE_CONV_BODY_AT)

    def _freeze_backbone(self, freeze_at):
        if freeze_at < 0:
            return
        for stage_index in range(freeze_at):
            if stage_index == 0:
                m = self.stem  # stage 0 is the stem
            else:
                m = getattr(self, "layer" + str(stage_index))
            for p in m.parameters():
                p.requires_grad = False

    def forward(self, x):
        start_timer=time.time()
        outputs = []
        x = self.stem(x)
        for stage_name in self.stages:
            x = getattr(self, stage_name)(x)
            if self.return_features[stage_name]:
                outputs.append(x)
        print("ResNet time :: ", time.time()-start_timer,file=open("timelogger.log","a"))
        return outputs

Only change that has to be made is in the forward pass and all the instance created out of this class will inherit the properties and log time (choose to write the same to the file instead of a simple stdout)


Instruction: How to retrieve topk values across multiple rows along with their respective indices in PyTorch?

Input:
I have a tensor like this:

tensor([[[ 7.3478, -1.8058, -2.6140,  ..., -0.2719, -0.3171, -0.4737]],

    [[ 7.3606, -1.8269, -1.9825,  ..., -0.8680,  0.4894,  0.2708]]],
   grad_fn=<CatBackward>)

I want to get the topk values across both the rows. Currently what I am able to do is the following:

ipdb>  stacked.topk(2)
torch.return_types.topk(
values=tensor([[[14.3902, 14.3039]],

        [[14.8927, 12.1973]]], grad_fn=<TopkBackward>),
indices=tensor([[[60, 12]],

        [[12, 23]]]))

From the output, you can see that the top 2 values were retrieved from both rows. I want to get an output as follows:

14.8927 that maps to index 12
14.3902 that maps to index 60

Note that if the top 2 values were in the first row, it will only return the values from there and completely ignore the second row and vice versa.

Need help in this regard.

A very hacky way of doing what I am trying to say would be the following but it is very hacky and shown for a BEAM_WIDTH of 2:

BEAM_WIDTH = 2
top_k = stacked.data.topk(BEAM_WIDTH, dim=2)

v1, i1 = top_k[0][0][0], top_k[1][0][0]
v2, i2 = top_k[0][1][0], top_k[1][1][0]

i = j = 0
final = []
for _ in range(BEAM_WIDTH):
    if v1[i] >= v2[j]:
        final.append((v1[i], i1[i]))
        i += 1
    else:
        final.append((v2[j], i2[j]))
        j += 1

Output:
Repeated Indices

I believe this is what you want. First you would find the topk elements in the flattened list, then convert those indices back to the row-relative format.

topk_values, linear_indices = stacked.flatten().topk(2)
topk_indices = linear_indices % stacked.shape[-1]

Unique Indices

The previous approach doesn't enforce unique indices. If unique indices are needed then you could find the max between rows, then find the topk among that.

topk_values, topk_indices = stacked.max(dim=0)[0].flatten().topk(2)

Example

To demonstrate the difference between these two approaches, suppose you have

stacked = torch.tensor([[[11,8,0]],
                        [[10,9,0]]])

In the repeated indices case you would end up with

topk_values=[11, 10]
topk_indices=[0, 0]

In the unique indices case you would get

topk_values=[11, 9]
topk_indices=[0, 1]


Instruction: How Can I Install pytorch on Intel & python2.7?

Input:
I often saw how to install pytorch on intel & python3
but no python2...!

How can I install pytorch on intel & python2

Output:
You can install PyTorch in two ways

Using pip and
Using conda

PyTorch installation using pip

pip install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html

PyTorch installation via conda

conda install pytorch torchvision cpuonly -c pytorch


Instruction: Pytorch specific operation for the finding dimension wise mean for a list of tensors

Input:
I have a list of tensors, each of 300x1 dimensions and I need to find the mean and variance for each dimension based on the list. How exactly this be achieved?. I could make this work in numpy with numpy arrays, with the following code snippet:

for item in embeddingLists:    # embeddingLists is a list of lists
                               # item contains list of numpy arrays
    tempVal =  np.mean(np.array(item),axis=0)
    meanVects.append(tempVal)

    temVar = np.var(np.array(item),axis=0,ddof=1)
    varVects.append(temVar)

Output:
With some help from pytorch discussion forum, I could solve the problem. Link to discussion

The relevant code is:

for item in embeddingLists:
    tempItem = [stuff.unsqueeze(0)  for stuff in item] #convert a 1x300 tensor
    coomn = torch.cat(tempItem)     # Convert to a 12x300 tensor
    temMean = torch.mean(coomn,dim=0)
    meanVects.append(temMean)
    temVar = torch.var(coomn,dim=0)
    varVects.append(temVar)


Instruction: Pytorch:Network not learning at all + Weights are too low

Input:
About the input. Sorry for the bad formatting. The for each two rows first row is the key and second row is the value. 18~20_ride is the label and is not included in the input. Below is one input. And train set consists of 400000 of these.

bus_route_id    station_code    latitude    longitude   6~7_ride
0               4270000         344         33.48990    126.49373
7~8_ride    8~9_ride    9~10_ride   10~11_ride  11~12_ride  6~7_takeoff
0.0         1.0         2.0         5.0         2.0         6.0
7~8_takeoff 8~9_takeoff 9~10_takeoff    10~11_takeoff   11~12_takeoff
0.0         0.0         0.0             0.0             0.0
18~20_ride  weekday dis_jejusi  dis_seoquipo
0.0         6       2.954920    26.256744

Example weights: Captured at 4th epoch. After 20 epochs of training I got much smaller values (ex. -7e-44 or 1e-55)

 2.3937e-11, -2.6920e-12, -1.0445e-11,  ..., -1.0754e-11, 1.1128e-11, -1.4814e-11

The model's prediction and target

#Target
[2.],
[0.],
[0.]

#Prediction
[1.4187],
[1.4187],
[1.4187]

MyDataset.py

from torch.utils.data import Dataset, DataLoader
import numpy as np
import pandas as pd
import torch
import os

class MyDataset(Dataset):
  def __init__(self, csv_filename):
    self.dataset = pd.read_csv(csv_filename, index_col=0)
    self.labels = self.dataset.pop("18~20_ride")
    self.dataset = self.dataset.values
    self.labels = np.reshape(self.labels.values,(-1,1))

  def __len__(self):
    return len(self.dataset)

  def __getitem__(self, idx):
    return self.dataset[idx], self.labels[idx]

Model

class Network(nn.Module):
    def __init__(self, input_num):
        super(Network, self).__init__()
        self.fc1 = nn.Sequential(
          nn.Linear(input_num, 64),
          nn.BatchNorm1d(64),
          GELU()
        )

        self.fc2 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc3 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc4 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc5 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc6 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU)
        )
        self.fc7 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc8 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU())
        )
        self.fc9 = nn.Linear(64, 1)

The training and validation

def train(model, device, train_loader, optimizer, loss_fn, log_interval, epoch):
  print("Training")
  model.train()
  for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.float().to(device), target.float().to(device)
    optimizer.zero_grad()
    output = model(data)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()
    if batch_idx % log_interval == 0:
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
            epoch, (batch_idx+1) * len(data), len(train_loader.dataset),
            100. * batch_idx / len(train_loader), loss.item()))

def validate(model, device, loader, loss_fn):
  print("\nValidating")
  model.eval()
  test_loss = 0
  correct = 0
  with torch.no_grad():
    for batch_idx, (data, target) in enumerate(loader):
      data, target = data.float().to(device), target.float().to(device)
      output = model(data)
      test_loss += loss_fn(output, target).item()  # sum up batch loss

  test_loss /= len(loader)

  print('Validation average loss: {:.4f}\n'.format(
      test_loss))
  return test_loss

Entire process of training and validation

from MyDataset import MyDataset
from torch.utils.data import DataLoader
from torch.optim.lr_scheduler import StepLR
from datetime import datetime

train_dataset_path = "/content/drive/My Drive/root/bus/dataset/train.csv"
val_dataset_path = "/content/drive/My Drive/root/bus/dataset/val.csv"
model_base_path = "/content/drive/My Drive/root/bus/models/"

model_file = "/content/drive/My Drive/root/bus/models/checkpoints/1574427776.202017.pt"

"""
Training Config
"""
epochs = 10
batch_size = 32
learning_rate = 0.5

check_interval = 4

log_interval = int(40000/batch_size)
gamma = 0.1

load_model = False
save_model = True
make_checkpoint = True
"""
End of config
"""

# Read test set
train_set = MyDataset(train_dataset_path)
train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True)
val_set = MyDataset(val_dataset_path)
val_loader = DataLoader(val_set, batch_size=1)
print("Data READY")

device = torch.device("cuda")
net = Network(19).float().to(device)
if load_model:
  net.load_state_dict(torch.load(model_file))
loss_fn = torch.nn.MSELoss()
optimizer = optim.AdamW(net.parameters(), lr=learning_rate)

best_loss = float('inf')
isAbort = False
for epoch in range(1, epochs+1):
  train(net, device, train_loader, optimizer, loss_fn, log_interval, epoch)
  val_loss = validate(net, device, val_loader, loss_fn)
  if epoch%check_interval==0:
    if make_checkpoint:
      print("Saving new checkpoint")
      torch.save(net.state_dict(), model_base_path+"checkpoints/"+str(datetime.today().timestamp())+".pt")
      """
  if val_loss < best_loss and epoch%check_interval==0:
    best_loss = val_loss
    if make_checkpoint:
      print("Saving new checkpoint")
      torch.save(net.state_dict(), model_base_path+"checkpoints/"+str(datetime.today().timestamp())+".pt")
  else:
    print("Model overfit detected. Aborting training")
    isAbort = True
    break
    """
if save_model and not isAbort:
    torch.save(net.state_dict(), model_base_path+"finals/"+str(datetime.today().timestamp())+".pt")

So I tried to train a fully connected model for a regression problem, with google colab. But it did not get trained well; The loss absolutely did not decrease. So I dug down and found out that the weights were really small. Any idea why this is happening and how I could avoid this? Thank you
I used MSE for loss and used ADaW optimizer. Below are the things I have tried

Tried other architectures (Changing number of layers sizes, Changed activation function ReLU, GELU)but the loss did not decrease
Tried changing the learning rate from 3e-1~1e-3, even tried 1
Tried other pre-processing(Used day/month/year instead of weekday) for the data
Given the label in the input data but loss did not decrease
Tried different batch_sizes(4, 10, 32, 64)
Removed batch_normalization
Other kinds of optimizer such as SGD, Adam
Training 20 epochs but loss did not decrease at all
The weights do change at loss.backward()

Output:
  TL;DR: Invalid input data!! Check for NaN or NULL

Well it has been sometime since the question. Tried almost everything and though maybe messed up the project setup. So I deleted the project and tried it again: same. Delete again and migrate to TF2: THE SAME RESULT! So I found out that there wasn't any problem with the setup. So I searched other places. In the end I did find the reason. The input columns were actually modified by myself. (To remove some highly correlated features). It was not original. During the modification I messed up some float values and it ended up having NaN values. So check if you're dataset contains invalid values.


Instruction: Cross validation for MNIST dataset with pytorch and sklearn

Input:
I am new to pytorch and are trying to implement a feed forward neural network to classify the mnist data set. I have some problems when trying to use cross-validation. My data has the following shapes:
 x_train:
torch.Size([45000, 784]) and
y_train: torch.Size([45000])

I tried to use KFold from sklearn.

kfold =KFold(n_splits=10)

Here is the first part of my train method where I'm dividing the data into folds:

for  train_index, test_index in kfold.split(x_train, y_train):
        x_train_fold = x_train[train_index]
        x_test_fold = x_test[test_index]
        y_train_fold = y_train[train_index]
        y_test_fold = y_test[test_index]
        print(x_train_fold.shape)
        for epoch in range(epochs):
         ...

The indices for the y_train_fold variable is right, it's simply:
[   0    1    2 ... 4497 4498 4499], but it's not for x_train_fold, which is [ 4500  4501  4502 ... 44997 44998 44999]. And the same goes for the test folds.

For the first iteration I want the varibale x_train_fold to be the first 4500 pictures, in other words to have the shape torch.Size([4500, 784]), but it has the shape torch.Size([40500, 784])

Any tips on how to get this right?

Output:
I think you're confused!

Ignore the second dimension for a while, When you've 45000 points, and you use 10 fold cross-validation, what's the size of each fold? 45000/10 i.e. 4500.

It means that each of your fold will contain 4500 data points, and one of those fold will be used for testing, and the remaining for training i.e.

  For testing: one fold => 4500 data points => size: 4500
  For training: remaining folds => 45000-4500 data points => size: 45000-4500=40500

Thus, for first iteration, the first 4500 data points (corresponding to indices) will be used for testing and the rest for training. (Check below image)

Given your data is x_train: torch.Size([45000, 784]) and y_train: torch.Size([45000]), this is how your code should look like:

for train_index, test_index in kfold.split(x_train, y_train):
    print(train_index, test_index)

    x_train_fold = x_train[train_index]
    y_train_fold = y_train[train_index]
    x_test_fold = x_train[test_index]
    y_test_fold = y_train[test_index]

    print(x_train_fold.shape, y_train_fold.shape)
    print(x_test_fold.shape, y_test_fold.shape)
    break

[ 4500  4501  4502 ... 44997 44998 44999] [   0    1    2 ... 4497 4498 4499]
torch.Size([40500, 784]) torch.Size([40500])
torch.Size([4500, 784]) torch.Size([4500])

So, when you say

  I want the variable x_train_fold to be the first 4500 picture... shape torch.Size([4500, 784]).

you're wrong. this size corresonds to x_test_fold. In the first iteration, based on 10 folds, x_train_fold will have 40500 points, thus its size is supposed to be torch.Size([40500, 784]).


Instruction: How to implement dropout in Pytorch, and where to apply it

Input:
I am quite unsure whether this is correct. It is really sad I can't find many good examples on how to parametrize a NN.

What do you think of this way of dropping out in those two classes. First I'm writing the original class :

class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

and then here, I found two different ways to write things, which I don't know how to distinguish. The first one uses :

self.drop_layer = nn.Dropout(p=p)

whereas the second :

self.dropout = nn.Dropout(p)

and here is my result :

class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.drop_layer = nn.Dropout(p=p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

 class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.dropout = nn.Dropout(p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

could this work, if not how to improve that, and is it giving me the result I m expecting, meaning creating a neural network where I can dropout some neurons. Important detail, I only want to dropout the second layer of neural network, no touch to the rest!

Output:
The two examples you provided are exactly the same. self.drop_layer = nn.Dropout(p=p) and self.dropout = nn.Dropout(p) only differ because the authors assigned the layers to different variable names. The dropout layer is typically defined in the .__init__() method, and called in .forward(). Like this:
 class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.dropout = nn.Dropout(p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.dropout(self.fc3(out))
      return out

You can do the test:
import torch
import torch.nn  as nn

m = nn.Dropout(p=0.5)
input = torch.randn(20, 16)
print(torch.sum(torch.nonzero(input)))
print(torch.sum(torch.nonzero(m(input))))

tensor(5440) # sum of nonzero values
tensor(2656) # sum on nonzero values after dropout

Let's visualize it:
import torch
import torch.nn as nn
input = torch.randn(5, 5)
print(input)

tensor([[ 1.1404,  0.2102, -0.1237,  0.4240,  0.0174],
        [-2.0872,  1.2790,  0.7804, -0.0962, -0.9730],
        [ 0.4788, -1.3408,  0.0483,  2.4125, -1.2463],
        [ 1.5761,  0.3592,  0.2302,  1.3980,  0.0154],
        [-0.4308,  0.2484,  0.8584,  0.1689, -1.3607]])

Now, let's apply the dropout:
m = nn.Dropout(p=0.5)
output = m(input)
print(output)

tensor([[ 0.0000,  0.0000, -0.0000,  0.8481,  0.0000],
        [-0.0000,  0.0000,  1.5608, -0.0000, -1.9459],
        [ 0.0000, -0.0000,  0.0000,  0.0000, -0.0000],
        [ 0.0000,  0.7184,  0.4604,  2.7959,  0.0308],
        [-0.0000,  0.0000,  0.0000,  0.0000, -0.0000]])

Approximately half the neurons have been turned to zero, because we had probability p=0.5 that a neuron is set to zero!


Instruction: Why is the Pytorch Dropout layer affecting all values, not only the ones set to zero?

Input:
The dropout layer from Pytorch changes the values that are not set to zero. Using Pytorch's documentation example: (source):

import torch
import torch.nn  as nn

m = nn.Dropout(p=0.5)
input = torch.ones(5, 5)

print(input)
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])

Then I pass it through a dropout layer:

output = m(input)
print(output)
tensor([[0., 0., 2., 2., 0.],
        [2., 0., 2., 0., 0.],
        [0., 0., 0., 0., 2.],
        [2., 2., 2., 2., 2.],
        [2., 0., 0., 0., 2.]])

The values that aren't set to zero are now 2. Why?

Output:
It is how the dropout regularization works. After a dropout the values are divided by the keeping probability (in this case 0.5).

Since PyTorch Dropout function receives the probability of zeroing a neuron as input, if you use nn.Dropout(p=0.2) that means it has 0.8 chance of keeping. so the values on the table will be 1/(1-0.2).

This is called "inverted dropout technique" and it is done in order to ensure that the expected value of the activation remains the same.


Instruction: How to Get n-Number of Images from torchvision.datasets.ImageFolder

Input:
I am currently experimenting on CNN using PyTorch, and the task I want the model to accomplish is to classify images.

I understand that using torchvision.datasets.ImageFolder can help with loading all images from my training folder, according to each subfolders' names as the labels.

I plan on taking only n-number of images from the ImageFolder randomly, but as far as I know, there is no mechanism for ImageFolder to load n-number of images randomly, where n is any number ranging from 1 to all available images.

How can I do that? Thanks for any help

Output:
You can create a subset of ImageFolder using PyTorch's Subset class. We can generate random indices using numpy or some other way if you want.

dataset = torchvision.datasets.ImageFolder(...)
dataset_subset = torch.utils.data.Subset(dataset, numpy.random.choice(len(dataset), n, replace=False))


Instruction: How can I implement a checkerboard stride for Conv2d in pytorch?

Input: I am trying to create a convnet using pytorch to work on an input of 2d matrices. I am using a 3x5 filter and I want it to have a custom stride as follows - on even line numbers I want the filter to start from the element at position 0 (red in the image), on odd line numbers I want it to start on the element of position 1 (blue in the image), and in both cases have a stride of 2 on the x direction. That means that if I have a matrix as in the image as my input, I want the filter to have only 0s in its center. I know this is very unusual in convnets but this is actually a problem in physics so the exact stride is important.

Output:
The following custom conv2d layer implements convolutions in a checkerboard stride as indicated in the original question. The difficulty here lies with the fact that pytorch doesn't really support inconsistent strides like this. That said we can break this operation into two separate strided convolutions, one for the even rows, and one for the odd rows. After that we can just interleave the results back together. There are some details in the code below which ensure we pad correctly (if desired). Also, this layer fully supports back-propagation.

import torch.nn as nn
import torch.nn.functional as F

class AMNI_Conv2d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, padding=0, bias=True):
        super().__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, bias=bias, padding=padding)
        self.crow = self.conv.kernel_size[0] // 2
        self.ccol = self.conv.kernel_size[1] // 2

        # this module only works with odd sized kernels
        assert self.conv.kernel_size[0] % 2 == 1 and self.conv.kernel_size[1] % 2 == 1

    def forward(self, x):
        # currently only padding with zeros is supported
        if self.conv.padding[0] != 0 or self.conv.padding[1] != 0:
            x = F.pad(x, pad=(self.conv.padding[1], self.conv.padding[1], self.conv.padding[0], self.conv.padding[0]))

        # center filters on the "zeros" according to the diagram by AMNI, starting column for even/odd rows may need to change depending on padding/kernel size
        if (self.crow + self.ccol + self.conv.padding[0] + self.conv.padding[1]) % 2 == 0:
            x_even = F.conv2d(x[:, :, :-1, 1:], self.conv.weight, self.conv.bias, stride=2)
            x_odd = F.conv2d(x[:, :, 1:, :-1], self.conv.weight, self.conv.bias, stride=2)
        else:
            x_even = F.conv2d(x[:, :, :-1, :-1], self.conv.weight, self.conv.bias, stride=2)
            x_odd = F.conv2d(x[:, :, 1:, 1:], self.conv.weight, self.conv.bias, stride=2)
        b, c, h, w = x_even.shape

        # interleave even and odd rows back together
        return torch.stack((x_even, x_odd), dim=3).contiguous().view(b, c, -1, w)

Example

This layer basically acts like a normal Conv2d but with the checkerboard stride.

>>> x = torch.arange(64).view(1, 1, 8, 8).float()
tensor([[[[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.],
          [ 8.,  9., 10., 11., 12., 13., 14., 15.],
          [16., 17., 18., 19., 20., 21., 22., 23.],
          [24., 25., 26., 27., 28., 29., 30., 31.],
          [32., 33., 34., 35., 36., 37., 38., 39.],
          [40., 41., 42., 43., 44., 45., 46., 47.],
          [48., 49., 50., 51., 52., 53., 54., 55.],
          [56., 57., 58., 59., 60., 61., 62., 63.]]]])

>>> layer = AMNI_Conv2d(1, 1, (3, 5), bias=False)

# set kernels to delta functions to demonstrate kernel centers
>>> with torch.no_grad():
...     layer.conv.weight.zero_()
...     layer.conv.weight[:,:,1,2] = 1

>>> result = layer(x)
tensor([[[[10., 12.],
          [19., 21.],
          [26., 28.],
          [35., 37.],
          [42., 44.],
          [51., 53.]]]], grad_fn=<ViewBackward>)

You could also do this with padding to get every "zero" in the original diagram

>>> layer = AMNI_Conv2d(1, 1, (3, 5), padding=(1, 2), bias=False)

# set kernels to delta functions to demonstrate kernel centers
>>> with torch.no_grad():
...     layer.conv.weight.zero_()
...     layer.conv.weight[:,:,1,2] = 1

>>> result = layer(x)
tensor([[[[ 1.,  3.,  5.,  7.],
          [ 8., 10., 12., 14.],
          [17., 19., 21., 23.],
          [24., 26., 28., 30.],
          [33., 35., 37., 39.],
          [40., 42., 44., 46.],
          [49., 51., 53., 55.],
          [56., 58., 60., 62.]]]], grad_fn=<ViewBackward>)


Instruction: Moving to numerically stable log-sum-exp leads to extremely large loss values

Input:
I am working on a network that uses a LSTM along with MDNs to predict some distributions. The loss function I use for these MDNs involve trying to fit my target data to the predicted distributions. I am trying to compute the log-sum-exp for the log_probs of these target data to compute the loss. When I use standard log-sum-exp, I get reasonable initial loss values (around 50-70) even though later it encounters some NaNs and breaks. Based on what I have read online, a numerically stable version of log-sum-exp is required to avoid this problem. However as soon I use the stable version, my loss values shoot up to the order of 15-20k. They do come down upon training but eventually they also lead to NaNs.

NOTE : I did not use the logsumexp function in PyTorch, since I needed to have a weighted summation based on my mixture components.

def log_sum_exp(self,value, weights, dim=None):
        eps = 1e-20
        m, idx = torch.max(value, dim=dim, keepdim=True)
        return m.squeeze(dim) + torch.log(torch.sum(torch.exp(value-m)*(weights.unsqueeze(2)),
                                       dim=dim) + eps)

def mdn_loss(self, pi, sigma, mu, target):
        eps = 1e-20
        target = target.unsqueeze(1)
        m = torch.distributions.Normal(loc=mu, scale=sigma)
        probs = m.log_prob(target)
        # Size of probs is batch_size x num_mixtures x num_out_features
        # Size of pi is batch_size x num_mixtures
        loss = -self.log_sum_exp(probs, pi, dim=1)
        return loss.mean()

Upon adding anomaly_detection, the NaNs seem to occur at :
        probs = m.log_prob(target)

Seeing these huge initial loss values just by moving to the numerically stable version have led me to believe I have some bug in my current implementation. Any help please.

Output: Issue resolved. My targets had some large values which were leading to overflow calculations when log_probs was calculated for these values. Removed some outlandish data points and normalised the data, loss immediately came down.


Instruction: Efficient way of selectively replacing vectors from a tensor in pytorch

Input:
Given a batch of text sequences, the same is converted into a tensor with each word represented using word embeddings or vectors (of 300 dimensions). I need to selectively replace vectors for certain specific words with a new set of embeddings. Further, this replacement will occur only for not all occurrences of the specific word, but only randomly. Currently, I have the following code to achieve this. It traverses through every word using 2 for loops, check if the word is in a specified list, splIndices. Then it checks if the word needs to be replaced or not, based on T or F value in selected_.

But could this be done in a more efficient manner?

The below code may not be an MWE, but I have tried to simplify the code by removing the specifics, so as to focus on the problem. Please ignore the semantics or purpose of the code as it may not have been appropriately represented in this snippet. The question is about improving performance.

splIndices = [45, 62, 2983, 456, 762]  # vocabulary indices which needs to be replaced
splFreqs = 2000  # assuming the words in splIndices occurs 2000 times
selected_ = Torch.Tensor(2000).uniform_(0, 1) > 0.2  # Tensor with 20% of the entries True
replIndexCtr = 0  # counter for selected_

# Dictionary with vectors to be replaced. This is a dummy function.
# Original function depends on some property of the word
diffVector = {45: Torch.Tensor(300).uniform_(0, 1), ...... 762: Torch.Tensor(300).uniform_(0, 1) }

embeding = nn.Embedding.from_pretrained(embedding_matrix, freeze=False)
tempVals = x  # shape [32, 41] - batch of 32 sequences with 41 words each
x = embeding(x) # shape [32, 41, 300] - the sequence now has replaced vocab indices with embeddings

# iterate through batch for sequences
for i, item in enumerate(x):
    # iterate sequences for words
    for j, stuff in enumerate(item):
        if tempVals[i][j].item() in splIndices:
            if self.selected_[replIndexCtr] == True:
                x[i,j] = diffVector[tempVals[i][j].item()]
                replIndexCtr += 1

Output:
It could be vectorized in a following way:

import torch
import torch.nn as nn
import torch.nn.functional as F

batch_size, sentence_size, vocab_size, emb_size = 3, 2, 15, 1

# Make certain bias as a marker of embedding
embedder_1 = nn.Linear(vocab_size, emb_size)
embedder_1.weight.data.fill_(0)
embedder_1.bias.data.fill_(200)

embedder_2 = nn.Linear(vocab_size, emb_size)
embedder_2.weight.data.fill_(0)
embedder_2.bias.data.fill_(404)

# Here are the indices of words which need different embdedding
replace_list = [3, 5, 7, 9]

# Make a binary mask highlighing special words' indices
mask = torch.zeros(batch_size, sentence_size, vocab_size)
mask[..., replace_list] = 1

# Make random dataset
data_indices = torch.randint(0, vocab_size, (batch_size, sentence_size))
data_onehot = F.one_hot(data_indices, vocab_size)

# Check if onehot of a word collides with replace mask
replace_mask = mask.long() * data_onehot
replace_mask = torch.sum(replace_mask, dim=-1).byte() # byte() is critical here

data_emb = torch.empty(batch_size, sentence_size, emb_size)

# Fill default embeddings
data_emb[1-replace_mask] = embedder_1(data_onehot[1-replace_mask].float())
if torch.max(replace_mask) != 0: # If not all zeros
    # Fill special embeddings
    data_emb[replace_mask] = embedder_2(data_onehot[replace_mask].float())

print(data_indices)
print(replace_mask)
print(data_emb.squeeze(-1).int())

Here is an example of a possible output:

# Word indices
tensor([[ 6,  9],
        [ 5, 10],
        [ 4, 11]])
# Embedding replacement mask
tensor([[0, 1],
        [1, 0],
        [0, 0]], dtype=torch.uint8)
# Resulting replacement
tensor([[200, 404],
        [404, 200],
        [200, 200]], dtype=torch.int32)


Instruction: How to get a particular layer output of a pretrained VGG16 in pytorch

Input:
I am very new to pytorch and I am trying to get the output of the pretrained model VGG16 feature vector in 1*4096 format which is returned by the layers just before the final layer. I found that there are similar features available in keras. Is there any direct command in pytorch for the same?

The code I am using:

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import models
from torch.autograd import Variable
from PIL import Image

image1 = Image.open(r"C:\Users\user\Pictures\user.png")

model = models.vgg16(pretrained=True)

scaler = transforms.Resize((224, 224))
to_tensor = transforms.ToTensor()

img = to_tensor(scaler(image1)).unsqueeze(0)

model(img).shape
model(img)

Output:
Part of the network responsible for creating features is named... features (not only in VGG, it's like that for most of the pretrained networks inside torchvision).

Just use this field and pass your image like this:

import torch

import torchvision

image = Image.open(r"C:\Users\user\Pictures\user.png")

# Get features part of the network
model = models.vgg16(pretrained=True).features
tensor = transforms.ToTensor()(transforms.Resize((224, 224))(image)).unsqueeze(dim=0)

model(tensor)

EDIT:

To see what happens inside any torchvision model you can check it's source code. For VGG (any), there is a base class at the top of this file.

To get 4096 flattened features, you could operations similar to those defined in forward:

# Tensor from previous code snippet for brevity

x = model.avgpool(tensor)
x = torch.flatten(x, 1)
final_x = model.classifier[0](x) # only first classifier layer

You could also iterate over modules or children up to wherever you want and output the result (or results or however you want)


Instruction: Debugging neural network dropout problem for the probability not lying inside [0,1]

Input:
I tried to put a droprate to my neural network (NN) using torch and I got a strange error at the end. How can I fix it?

So the idea is that I wrote a NN inside a function to make it easier to call. The function is the following:
(I personally think the problem lies inside the class of the NN, but for the sake of having a working example I'm putting everything).

def train_neural_network(data_train_X, data_train_Y, batch_size, learning_rate, graph = True, dropout = 0.0 ):
  input_size = len(data_test_X.columns)
  hidden_size = 200
  num_classes = 4
  num_epochs = 120
  batch_size = batch_size
  learning_rate = learning_rate

  # The class of NN
  class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p = dropout):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.fc3 = nn.Linear(hidden_size, num_classes)

    def forward(self, x, p = dropout):
          out = F.relu(self.fc1(x))
          out = F.relu(self.fc2(out))
          out = nn.Dropout(out, p) #drop
          out = self.fc3(out)
          return out

  # Prepare data
  X_train = torch.from_numpy(data_train_X.values).float()
  Y_train = torch.from_numpy(data_train_Y.values).float()

  # Loading data
  train = torch.utils.data.TensorDataset(X_train, Y_train)
  train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size)

  net = NeuralNet(input_size, hidden_size, num_classes)

  # Loss
  criterion = nn.CrossEntropyLoss()

  # Optimiser
  optimiser = torch.optim.SGD(net.parameters(), lr=learning_rate)

  # Proper training
  total_step = len(train_loader)
  loss_values = []

  for epoch in range(num_epochs+1):
    net.train()

    train_loss = 0.0

    for i, (predictors, results) in enumerate(train_loader, 0):
      # Forward pass
      outputs = net(predictors)
      results = results.long()
      results = results.squeeze_()
      loss = criterion(outputs, results)

      # Backward and optimise
      optimiser.zero_grad()
      loss.backward()
      optimiser.step()

      # Update loss
      train_loss += loss.item()

    loss_values.append(train_loss / batch_size )
  print('Finished Training')

  return net

And when I call the function:

net = train_neural_network(data_train_X = data_train_X, data_train_Y = data_train_Y, batch_size = batch_size, learning_rate = learning_rate, dropout = 0.1)

The error is the following:

net = train_neural_network(data_train_X = data_train_X, data_train_Y = data_train_Y, batch_size = batch_size, learning_rate = learning_rate, dropout = 0.1)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/dropout.py in __init__(self, p, inplace)
      8     def __init__(self, p=0.5, inplace=False):
      9         super(_DropoutNd, self).__init__()
---> 10         if p < 0 or p > 1:
     11             raise ValueError("dropout probability has to be between 0 and 1, "
     12                              "but got {}".format(p))

RuntimeError: bool value of Tensor with more than one value is ambiguous

Why do you think there is an error?

Before putting the droprate, everything was working. Additional points for you if you know how to
implement a bias inside my network! For example, on the hidden layer. I can't find any example online.

Output:
Change your architecture for this:

class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p=dropout):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.fc3 = nn.Linear(hidden_size, num_classes)
        self.dropout = nn.Dropout(p=p)

    def forward(self, x):
        out = F.relu(self.fc1(x))
        out = F.relu(self.fc2(out))
        out = self.dropout(self.fc3(out))
        return out

Let me know if it works.


Instruction: Run-time error for (allegedly) empty batches in Pytorch

Input:
(Updated with further insights on the problem)

I have a dataset of 3000 images which gets into a DataLoader with the following lines:

training_left_eyes = torch.utils.data.DataLoader(train_dataset, batch_size=2,shuffle=True, drop_last=True)
print(len(training_left_eyes)) #Outputs 1500

My training loop looks like this:

for i,(data,output) in enumerate(training_left_eyes):
      data,output = data.to(device),output.to(device)
      prediction = net(data)

      loss = costFunc(prediction,output)
      closs = loss.item()
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()

      print("batch #{}".format(i))
      if i%100 == 0:
        print('[%d %d] loss: %.8f' % (epoch+1,i+1,closs/1000))
        closs = 0

The information inside the tensors "data" and "output" (labels) are correct and the system works fine until it reaches the batch number 1500. All my batches are full as 3000/2=1500 with no remainder. Once it reaches that last batch there is a RunTimeError stating that there is a 0-dimensional input size. But I don't know why this is supposed to happen as enumerate(training_left_eyes) should be iterating over the values of the DataLoader which are full.

I searched on the web how to solve this and some people were mentioning the "drop_last=True" attribute on the DataLoader, althought this was done in order to prevent semi-empty batches to enter the model I tried it anyway without no avail.

I am starting to tunnel vision too much and can't seem to solve the problem on my own. I could simply insert an if statement but I think that would be very bad practice and I want to learn the proper solution.

If it helps, here is my custom DataSet:

class LeftEyeDataset(torch.utils.data.Dataset):
  """Left eye retinography dataset. Normal/Not-normal"""

  def __init__(self, csv_file, root_dir, transform=None):
    """
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    self.labels  = label_mapping(csv_file)
    self.root_dir = root_dir
    self.transform = transform
    self.names = name_mapping(csv_file)

  def __len__(self):
    return len(self.labels)

  def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    img_name = self.root_dir +'/'+ self.names[idx]
    image = io.imread(img_name)
    label = self.labels[idx]

    if self.transform:
        image = self.transform(image)

    return image,label

def label_mapping(csv_file) -> np.array:
  df = read_excel(excel_file, 'Sheet1')
  x= []
  for key,value in df['Left-Diagnostic Keywords'].iteritems():
    if value=='normal fundus':
      x.append(1)
    else:
      x.append(0)
  x_tensor = torch.LongTensor(x)
  return x_tensor

def name_mapping(csv_file) -> list:
  #Reads the names of the excel file
  df = read_excel(excel_file, 'Sheet1')
  names= list()
  serie = df['Left-Fundus']
  for i in range(serie.size):
    names.append(df['Left-Fundus'][i])
  return names

I can provide any additional code if needed.

Update: after some time trying to solve the problem I manage to pin point what is happening. For some reason on the last batch, the data entering the network is fine but right before the first layer something happens and it dissapears. On this next image you can see a print I do before entering the forward(self,x) and one right after. The sizes are aligned until the batch number 61 (I reduced it from 1500 for this example) in which it somehow goes through the print twice. After that line, the error mentioned above appears.

Output:
After rubber duck debugging for a bit I realized that the problem did not relied in the training but on the validation set. The code read

for i,(data,output) in validate_left_eyes:
      data,output = data.to(device),output.to(device)
      prediction = net(data)

Validate_left_eyes_ was not wrapped by an enumerate() and thus the first batch of data was empty. The problem was then solved.

I apologize as this part of the code was not featured on my question and thus the answer was not so straight forward.


Instruction: RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Input:
This:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

for data in dataloader:
    inputs, labels = data
    outputs = model(inputs)

Gives the error:

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Output:
You get this error because your model is on the GPU, but your data is on the CPU. So, you need to send your input tensors to the GPU.
inputs, labels = data                         # this is what you had
inputs, labels = inputs.cuda(), labels.cuda() # add this line

Or like this, to stay consistent with the rest of your code:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

inputs, labels = inputs.to(device), labels.to(device)

The same error will be raised if your input tensors are on the GPU but your model weights aren't. In this case, you need to send your model weights to the GPU.
model = MyModel()

if torch.cuda.is_available():
    model.cuda()

See the documentation for cuda(), and its opposite, cpu().


Instruction: SyntaxError: only named arguments may follow *expression on nn.Sequential()

Input:
def forward(self, x):
    x = x.view(-1)
    fwd = nn.Sequential(self.linear_1, *self.middle, self.output)
    return fwd(x)

It works python3 but not python2.7. The error is:

  fwd = nn.Sequential(self.linear_1, *self.middle, self.output)
  SyntaxError: only named arguments may follow *expression
  )

What can I convert this code to work on python2.7?

Output:
You could try the following syntax:

fwd = nn.Sequential(self.linear_1, *(self.middle + [self.output]))

This is assuming that self.middle is a list while self.output is an item (of class nn.Module).

The trick here is to combine self.middle and self.output in a single list and pass that to the nn.Sequential method.


Instruction: Parameter-specific learning rate in PyTorch

Input:
How can I set a learning rate for each specific parameter (weights and biases) in a network?

On PyTorch's docs I found this:

optim.SGD([{'params': model.base.parameters()},
           {'params': model.classifier.parameters(), 'lr': 1e-3}],
           lr=1e-2, momentum=0.9)

where model.classifier.parameters(), which defines a group of parameters obtains a specific learning rate of 1e-3.

But how can I translate this into parameter level?

Output:
You can set parameter-specific learning rate by using the parameter names to set the learning rates e.g.

For a given network taken from PyTorch forum:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer1 = nn.Linear(1, 1)
        self.layer1.weight.data.fill_(1)
        self.layer1.bias.data.fill_(1)
        self.layer2 = nn.Linear(1, 1)
        self.layer2.weight.data.fill_(1)
        self.layer2.bias.data.fill_(1)

    def forward(self, x):
        x = self.layer1(x)
        return self.layer2(x)

net = Net()
for name, param in net.named_parameters():
    print(name)

The parameters are:

layer1.weight
layer1.bias
layer2.weight
layer2.bias

Then, you can use the parameter names to set their specific learning rates as follows:

optimizer = optim.Adam([
            {'params': net.layer1.weight},
            {'params': net.layer1.bias, 'lr': 0.01},
            {'params': net.layer2.weight, 'lr': 0.001}
        ], lr=0.1, weight_decay=0.0001)

out = net(torch.Tensor([[1]]))
out.backward()
optimizer.step()
print("weight", net.layer1.weight.data.numpy(), "grad", net.layer1.weight.grad.data.numpy())
print("bias", net.layer1.bias.data.numpy(), "grad", net.layer1.bias.grad.data.numpy())
print("weight", net.layer2.weight.data.numpy(), "grad", net.layer2.weight.grad.data.numpy())
print("bias", net.layer2.bias.data.numpy(), "grad", net.layer2.bias.grad.data.numpy())

Output:

weight [[0.9]] grad [[1.0001]]
bias [0.99] grad [1.0001]
weight [[0.999]] grad [[2.0001]]
bias [1.] grad [1.]


Instruction: How can I add bias using pytorch to a neural network?

Input:
If I have this neural network (NN):

 class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p=dropout):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.fc3 = nn.Linear(hidden_size, num_classes)
        self.dropout = nn.Dropout(p=p)

    def forward(self, x):
        out = F.relu(self.fc1(x))
        out = F.relu(  self.dropout(self.fc2(out)))
        out = self.fc3(out)
        return out

How do you add bias to it? For instance, on the hidden layer. I can't find anything online.

Output:
It depends on the layer you are using. Some do not have that option. In linear, for example, you can use:

self.fc1 = nn.Linear(input_size, hidden_size, bias =False) # Either true or false, the default is true.

In the documentation you can see for other types of layers if they have a bias option or not.