Instruction: How to manage the hidden state dims when using pad_sequence?

Input:
Using Pytorch LSTM architecture trying to build a text generation model. For every batch, I'm using pad_sequence to have min padding for every sequence, therefore I have a variable dims batch (batch_size * seq_len). I'm applying also pack_padded_seq to only give the non-zero (non-padding) tokens to the LSTM. But, the variable dims batch throws an error while feeding it to the LSTM as following; Expected hidden[0] size (1, 8, 16), got (1, 16, 16). In this error, I have provided batch size 16 with 8 tokens for every sequence, but the hidden state is 16 * 16.

I have tried to create the hidden state in the forward function, but that did not work well. How can I create the hidden state such that it can accept variable dims batch and it will not be lost for the whole epoche?

class RNNModule(nn.Module):
    def __init__(self, n_vocab, seq_size, embedding_size, lstm_size):
        super(RNNModule, self).__init__()
        self.seq_size = seq_size
        self.lstm_size = lstm_size
        self.embedding, num_embeddings, embedding_dim = create_emb_layer(weight_matrix, False)
        self.lstm = nn.LSTM(embedding_size,
                        lstm_size,
                        num_layers=flags.n_layers,
                        batch_first=True
                        )
        self.dense = nn.Linear(lstm_size, n_vocab)

    def forward(self, x,length,prev_state):

        embed = self.embedding(x)
        packed_input = db.pack_src(embed,length)
        packed_output, state = self.lstm(packed_input,prev_state)
        padded,_ = db.pad_pack(packed_output)
        logits = self.dense(padded)
        return logits, state

    def zero_state(self, batch_size = flags.batch_size):
        return (torch.zeros(flags.n_layers, batch_size, self.lstm_size),
            torch.zeros(flags.n_layers, batch_size, self.lstm_size))

input: tensor([[  19,    9,    4,    3,   68,    8,    6,    2],
    [  19,    9,    4,    3,    7,    8,    6,    2],
    [   3,   12,   17,   10,    6,   40,    2,    0],
    [   4,    3,  109,    7,    6,    2,    0,    0],
    [ 188,    6,    7,   18,    3,    2,    0,    0],
    [   4,    3,   12,    6,    7,    2,    0,    0],
    [   6,    7,    3,   13,    2,    0,    0,    0],
    [   3,   28,   17,   69,    2,    0,    0,    0],
    [   6,    3,   12,   11,    2,    0,    0,    0],
    [   3,   13,    6,    7,    2,    0,    0,    0],
    [   3,    6,    7,   13,    2,    0,    0,    0],
    [   6,    3,   23,    7,    2,    0,    0,    0],
    [   3,   28,   10,    2,    0,    0,    0,    0],
    [   6,    3,   23,    2,    0,    0,    0,    0],
    [   3,    6,   37,    2,    0,    0,    0,    0],
    [1218,    2,    0,    0,    0,    0,    0,    0]])

Zero tokens are padding.
Embedding size: 64
LSTM size: 16
batch size: 16

Output:
The size of the hidden state you create has the correct size, but your input does not. When you pack it with nn.utils.rnn.pack_padded_sequence you've set batch_first=False, but your data has size [batch_size, seq_len, embedding_size] when you pass it to the packing, so that has batch_size as the first dimension. Also for the LSTM you use batch_first=True, which is appropriate for your data.

You only need to pack it correctly by setting batch_first=True as well, to match the order of your data.

rnn_utils.pack_padded_sequence(embed,length,batch_first=True)


Instruction: Understanding the parameters of a simple neural network in Pytorch

Input:
I have defined a simple NN as follows in Pytorch:

my_model = nn.Sequential(
            nn.Linear(1, 5),
            nn.Tanh(),
            nn.Linear(5, 1))

I then iterate through the parameters and check their sizes:

[parameter.shape for parameter in my_model.parameters()]

I get:

[torch.Size([5, 1]), torch.Size([5]), torch.Size([1, 5]), torch.Size([1])]

I'm confused as to why the last Size is 1. Shouldn't there be 5 bias values as well, going out from the hidden layer into the output layer?

Output:
Um, so it looks like you are defining a two layers network.

The input size is [1], and the hidden layer size is [5], and next, the hidden layer is connected to the output layer, the output layer size is [1].

It should look like:

       *   <- input
   / / | \ \
   * * * * *  <- hidden layer
   \ \ | / /
       *   <- output

So for the hidden layer, there are 5 values as bias, for the output layer, there is only one value as bias. Sounds reasonable?


Instruction: regarding one code segment in computing log_sum_exp

Input:
In this tutorial on using Pytorch to implement BiLSTM-CRF, author implements the following function. In specific, I am not quite understand what does max_score_broadcast = max_score.view(1, -1).expand(1, vec.size()[1]) try to do?, or which kind of math formula it corresponds to?

# Compute log sum exp in a numerically stable way for the forward algorithm
def log_sum_exp(vec):
    max_score = vec[0, argmax(vec)]
    max_score_broadcast = max_score.view(1, -1).expand(1, vec.size()[1])
    return max_score + \
        torch.log(torch.sum(torch.exp(vec - max_score_broadcast)))

Output:
Looking at the code, it seems like vec has a shape of (1, n).
Now we can follow the code line by line:

max_score = vec[0, argmax(vec)]

Using vec in the location 0, argmax(v) is just a fancy way of taking the maximal value of vec. So, max_score is (as the name suggests) the maximal value of vec.

max_score_broadcast = max_score.view(1, -1).expand(1, vec.size()[1])

Next, we want to subtract max_score from each of the elements of vec.
To do so the code creates a vector of the same shape as vec with all elements equal to max_score.
First, max_score is reshaped to have two dimensions using the view command, then the expanded 2d vector is "stretched" to have length n using the expand command.

Finally, the log sum exp is computed robustly:

 return max_score + \
        torch.log(torch.sum(torch.exp(vec - max_score_broadcast)))

The validity of this computation can be seen in this picture:

The rationale behind it is that exp(x) can "explode" for x > 0, therefore, for numerical stability, it is best to subtract the maximal value before taking exp.

As a side note, I think a slightly more elegant way to do the same computation, taking advantage of broadcasting, would be

max_score, _ = vec.max(dim=1, keepdim=True)  # take max along second dimension
lse = max_score + torch.log(torch.sum(torch.exp(vec - max_score), dim=1))
return lse

Also note that log sum exp is already implemented by pytorch: torch.logsumexp.


Instruction: Custom Dataset, Dataloader, Sampler, or something else?

Input:
I'm working on a project that requires training a PyTorch framework NN on a very large dataset of images. Some of these images are completely irrelevant to the problem, and but these irrelevant images are not labelled as such. However, there are some metrics I can use to calculate if they are irrelevant (e.g. summing all the pixel values would give me a good sense of which are the relevant images and which are not). What I would ideally like to do is have a Dataloader that can take in a Dataset class, and create batches only with the relevant images. The Dataset class would just know the list of images and their labels, and the Dataloader would interpret whether or not the image it is making a batch with is relevant or not, and would then only make batches with relevant images.

To apply this to an example, lets say I have a dataset of black and white images. The white images are irrelevant, but they are not labelled as such. I want to be able to load batches from a file location, and have these batches only contain the black images. I could filter at some point by summing all the pixels and finding it equals to 0.

What I am wondering is if a custom Dataset, Dataloader, or Sampler would be able to solve this task for me? I already have written a custom Dataset that stores the directory of all the saved images, and a list of all the images in that directory, and can return an image with its label in the getitem function. Is there something more I should add there to filter out certain images? Or should that filter be applied in a custom Dataloader, or Sampler?

Thank you!

Output:
I'm assuming that your image dataset belongs to two classes (0 or 1) but it's unlabeled. As @PranayModukuru mentioned that you can determine the similarity by using some measure (e.g aggregating all the pixels intensity values of a image, as you mentioned) in the getitem function in tour custom Dataset class.

However, determining the similarity in getitem function while training your model will make the training process very slow. So, i would recommend you to approximate the similarity before start training (not in the getitem function). Moreover if your image dataset is comprised of complex images (not black and white images) it's better to use a pretrained deep learning model (e.g. resnet or autoencoder) for dimentionality reduction followed by applying clustering approach (e.g. agglomerative clustering) to label your image.

In the second approach you only need to label your images for exactly one time and if you apply augmentation on images while training you don't need to re-determine the similarity (label) in the getitem funcion. On the other hand, in the first approach you need to determine the similarity (label) every time (after applying transformation on images) in the getitem function which is redundant, unnecessary and time consuming.

Hope this will help.


Instruction: Why my accuracy values don't change on my train model

Input:
When I start to train my model, Loss values decreasing but Accuracy values never change.I don't know why?

# -*- coding: utf-8 -*-

#Libraries
import torch
import torch.nn.functional as F
from torch import autograd, nn
from torch.autograd import Variable
import numpy as np
import matplotlib.pyplot as plt
from torchvision import transforms, datasets
from torch.utils import data

"""
Olivetti face dataset
"""
from sklearn.datasets import fetch_olivetti_faces

# Olivetti dataset download
olivetti = fetch_olivetti_faces()
train = olivetti.images
label = olivetti.target

X = train
Y = label
print("Format for X:", X.shape)
print("Format for Y: ", Y.shape)

print("\nDownload Ok")

"""
Set for train
"""
train_rate = 0.8

X_train = np.zeros([int(train_rate * X.shape[0]),64,64], dtype=float)
Y_train = np.zeros([int(train_rate * X.shape[0])], dtype=int)

X_val = np.zeros([int((1-train_rate) * X.shape[0]+1),64,64], dtype=float)
Y_val = np.zeros([int((1-train_rate) * X.shape[0]+1)], dtype=int)

#Split data for train and validation
for i in range(X.shape[0]):
    ie=0
    iv=0
    if (i%10)/10 >= train_rate:
        X_train[ie] = X[i]
        Y_train[ie] = Y[i]
        ie += 1
    else:
        X_val[iv] = X[i]
        Y_val[iv] = Y[i]
        iv += 1

X_train = X_train.reshape(320,-1,64,64)
X_val = X_val.reshape(80,-1,64,64)

print(Y_train.shape)
X_train = torch.Tensor(X_train)
Y_train = torch.Tensor(Y_train)

X_val = torch.Tensor(X_val)
Y_val = torch.Tensor(Y_val)

batch_size = 20

train_loader = torch.utils.data.DataLoader(X_train,
                                            batch_size=batch_size,
                                            )
val_loader = torch.utils.data.DataLoader(X_val,
                                            batch_size=batch_size,
                                            )

class CNNModule(nn.Module):
    def __init__(self):
        super(CNNModule, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 13 * 13, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 40)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 13 * 13)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

def make_train(model,dataset,n_iters,gpu):

    # Organize data
    X_train,Y_train,X_val,Y_val = dataset

    kriter = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(),lr=0.01)

    #Arrays to save loss and accuracy
    tl=np.zeros(n_iters)    #For train loss
    ta=np.zeros(n_iters)    #For train accuracy
    vl=np.zeros(n_iters)    #For validation loss
    va=np.zeros(n_iters)    #For validation accuracy

    # Convert labels to long
    Y_train = Y_train.long()
    Y_val = Y_val.long()

    # GPU control
    if gpu:
        X_train,Y_train = X_train.cuda(),Y_train.cuda()
        X_val,Y_val = X_val.cuda(),Y_val.cuda()
        model = model.cuda() # Parameters to GPU!
        print("Using GPU")
    else:
        print("Using CPU")
        # print(X_train.shape)
        # print(Y_train.shape)

    for i in range(n_iters):

        # train forward
        train_out = model.forward(X_train)
        train_loss = kriter(train_out,Y_train)

        # Backward and optimization
        train_loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        # Compute train accuracy
        train_predict = train_out.cpu().detach().argmax(dim=1)
        train_accuracy = (train_predict.cpu().numpy()==Y_train.cpu().numpy()).mean()

        # For validation
        val_out = model.forward(X_val)
        val_loss = kriter(val_out,Y_val)

        # Compute validation accuracy
        val_predict = val_out.cpu().detach().argmax(dim=1)
        val_accuracy = (val_predict.cpu().numpy()==Y_val.cpu().numpy()).mean()

        tl[i] = train_loss.cpu().detach().numpy()
        ta[i] = train_accuracy
        vl[i] = val_loss.cpu().detach().numpy()
        va[i] = val_accuracy

        # Show result each 5 loop
        if i%5==0:
            print("Loop --> ",i)
            print("Train Loss :",train_loss.cpu().detach().numpy())
            print("Train Accuracy :",train_accuracy)
            print("Validation Loss :",val_loss.cpu().detach().numpy())
            print("Validation Accuracy :",val_accuracy)

    model = model.cpu()

    #Print result
    plt.subplot(2,2,1)
    plt.plot(np.arange(n_iters), tl, 'r-')
    plt.subplot(2,2,2)
    plt.plot(np.arange(n_iters), ta, 'b--')

    plt.subplot(2,2,3)
    plt.plot(np.arange(n_iters), vl, 'r-')
    plt.subplot(2,2,4)
    plt.plot(np.arange(n_iters), va, 'b--')

dataset = X_train,Y_train,X_val,Y_val

gpu =  True
gpu = gpu and torch.cuda.is_available()

model = CNNModule()
make_train(model,dataset,100,gpu)

OUTPUT:

Using CPU
Loop -->  0
Train Loss : 3.6302185
Train Accuracy : 0.0
Validation Loss : 3.6171098
Validation Accuracy : 0.0
Loop -->  5
Train Loss : 3.557933
Train Accuracy : 0.996875
Validation Loss : 3.545982
Validation Accuracy : 0.9875
.
.
.
Loop -->  95
Train Loss : 0.04211783
Train Accuracy : 0.996875
Validation Loss : 0.13397054
Validation Accuracy : 0.9875

Output:
From your code,

train_accuracy = (train_predict.cpu().numpy()==Y_train.cpu().numpy()).mean()
you are taking mean of correct values that's why you are getting same answer in every loop.  Instead you should divide the total correct numbers with total number of examples to find the accuracy.


Instruction: Python Dataset Class + PyTorch Dataloader: Stuck at __getitem__, how to get Index, Label and so on during Testing?

Input:
I have a, maybe small problem but I am stuck for quite a while now. Hope someone can help me with that. I am currently on a Kddcup99 dataset which I like to train via DeepLearning (CNN Network)

I have a "Dataset" Class which includes the Panda Dataframe. Thus i split up into normal and validate dataset. So far, no problem.
I load it into a Numpy vector, torch it to Tensor and then direct it to the DataLoader.

The Dataset Class has these two important classes for iterating through:

def __len__(self):
        return len(self.val_df)

def __getitem__(self, index):
        img, target = self.val_df[index][:-1], self.val_df[index][-1]
        return img, target, index

Not in the class is the DataLoader string:

test_dataloader = DataLoader(datat.val_df, batch_size=10, shuffle=True)

In my Trainer Class i have a for loop which should iterate through the Dataloader:

with torch.no_grad():
            for data in dataloader:
                inputs, labels, idx = data
                inputs = inputs.to(self.device)

But it won't. I can't access the labels, index and such.

My question is now: Why?
How can I access Labels, Index from the given Dataset via the Dataloader?

Thank you all for your help!
Much appreciate it.

Output:
The first argument to DataLoader is the dataset from which you want to load the data, that's usually a Dataset, but it's not restricted to any instance of Dataset. As long as it defines the length (__len__) and can be indexed (__getitem__ allows that) it is acceptable.

You are passing datat.val_df to the DataLoader, which is presumably a NumPy array. A NumPy array has a length and can be indexed, so it can be used in the DataLoader. Since you pass that array directly, your dataset's __getitem__ is never called, but the array itself is indexed, so every item is just data.val_df[index].

Instead of using the underlying data for the DataLoader, you have to use the dataset itself (datat):

test_dataloader = DataLoader(datat, batch_size=10, shuffle=True)


Instruction: PyTorch: Row-wise Dot Product

Input:
Suppose I have two tensors:

a = torch.randn(10, 1000, 1, 4)
b = torch.randn(10, 1000, 6, 4)

Where the third index is the index of a vector.

I want to take the dot product between each vector in b with respect to the vector in a.

To illustrate, this is what I mean:

dots = torch.Tensor(10, 1000, 6, 1)
for b in range(10):
     for c in range(1000):
           for v in range(6):
            dots[b,c,v] = torch.dot(b[b,c,v], a[b,c,0])

How would I achieve this using torch functions?

Output:
a = torch.randn(10, 1000, 1, 4)
b = torch.randn(10, 1000, 6, 4)

c = torch.sum(a * b, dim=-1)

print(c.shape)

torch.Size([10, 1000, 6])

c = c.unsqueeze(-1)
print(c.shape)

torch.Size([10, 1000, 6, 1])


Instruction: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 3-dimensional input of size [3, 32, 32] instead

Input:
I am new to PyTorch and neural networks in general. I was trying to implement the resnet-50 model from torchvision on the CIFAR-10 dataset.

import torchvision
import torch
import torch.nn as nn
from torch import optim
import os
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
import numpy as np
from collections import OrderedDict
import matplotlib.pyplot as plt

transformations=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])
trainset=torchvision.datasets.CIFAR10(root='./CIFAR10',download=True,transform=transformations,train=True)

testset=torchvision.datasets.CIFAR10(root='./CIFAR10',download=True,transform=transformations,train=False)

trainloader=DataLoader(dataset=trainset,batch_size=4)
testloader=DataLoader(dataset=testset,batch_size=4)

inputs,labels=next(iter(trainset))
inputs.size()
resnet=torchvision.models.resnet50(pretrained=True)

if torch.cuda.is_available():
  resnet=resnet.cuda()
  inputs,labels=inputs.cuda(),torch.Tensor(labels).cuda()

outputs=resnet(inputs)

OUTPUT

--------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-904acb410fe4> in <module>()
      6   inputs,labels=inputs.cuda(),torch.Tensor(labels).cuda()
      7
----> 8 outputs=resnet(inputs)

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    344                             _pair(0), self.dilation, self.groups)
    345         return F.conv2d(input, weight, self.bias, self.stride,
--> 346                         self.padding, self.dilation, self.groups)
    347
    348     def forward(self, input):

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 3-dimensional input of size [3, 32, 32] instead

Is there a problen with the dataset for some reason and if not, how do i give a 4 dimensional input? Is the torchvision implementation of ResNet-50 not usable for CIFAR-10?

Output:
Currently you are iterating over dataset that's why you are getting a (3-dimensional) single image. You actually need to iterate over dataloader to get a 4-dimensional image batch. Therefore, you just need to change the following line:

inputs,labels=next(iter(trainset))

to

inputs,labels=next(iter(trainloader))


Instruction: How to pass an intermediate layer of one model to another model for skip connection in PyTorch

Input:
I want to define an encoder decoder architecture as two separate models and later connect them using nn.Sequential() as shown in the code below. Now, let's say that I want to connect/concatenate the output of Encoder conv4 block to the deconv1 block of Decoder as a skip connection. Is there a way to achieve that without combining the two models (encoder and decoder) into one. I want to keep them separate to be able to use output of the same encoder as input of multiple decoders.

class Encoder(nn.Module):

    def __init__(self, conv_dim=64, n_res_blocks=2):
        super(Encoder, self).__init__()

        # Define the encoder
        self.conv1 = conv(3, conv_dim, 4)
        self.conv2 = conv(conv_dim, conv_dim*2, 4)
        self.conv3 = conv(conv_dim*2, conv_dim*4, 4)
        self.conv4 = conv(conv_dim*4, conv_dim*4, 4)

        # Define the resnet part of the encoder
        # Residual blocks
        res_layers = []
        for layer in range(n_res_blocks):
            res_layers.append(ResidualBlock(conv_dim*4))
        # use sequential to create these layers
        self.res_blocks = nn.Sequential(*res_layers)

        # leaky relu function
        self.leaky_relu = nn.LeakyReLU(negative_slope=0.2)

    def forward(self, x):
        # define feedforward behavior, applying activations as necessary
        conv1 = self.leaky_relu(self.conv1(x))
        conv2 = self.leaky_relu(self.conv2(conv1))
        conv3 = self.leaky_relu(self.conv3(conv2))
        conv4 = self.leaky_relu(self.conv4(conv3))

        out = self.res_blocks(conv4)

        return out

# Define the Decoder Architecture
class Decoder(nn.Module):

    def __init__(self, conv_dim=64, n_res_blocks=2):
        super(Decoder, self).__init__()

        # Define the resnet part of the decoder
        # Residual blocks
        res_layers = []
        for layer in range(n_res_blocks):
            res_layers.append(ResidualBlock(conv_dim*4))
        # use sequential to create these layers
        self.res_blocks = nn.Sequential(*res_layers)

        # Define the decoder
        self.deconv1 = deconv(conv_dim*4, conv_dim*4, 4)
        self.deconv2 = deconv(conv_dim*4, conv_dim*2, 4)
        self.deconv3 = deconv(conv_dim*2, conv_dim, 4)
        self.deconv4 = deconv(conv_dim, conv_dim, 4)

        # no batch norm on last layer
        self.out_layer = deconv(conv_dim, 3, 1, stride=1, padding=0, normalization=False)

        # leaky relu function
        self.leaky_relu = nn.LeakyReLU(negative_slope=0.2)

    def forward(self, x):
        # define feedforward behavior, applying activations as necessary
        res = self.res_blocks(x)

        deconv1 = self.leaky_relu(self.deconv1(res))
        deconv2 = self.leaky_relu(self.deconv2(deconv1))
        deconv3 = self.leaky_relu(self.deconv3(deconv2))
        deconv4 = self.leaky_relu(self.deconv4(deconv3))

        # tanh applied to last layer
        out = F.tanh(self.out_layer(deconv4))
        out = torch.clamp(out, min=-0.5, max=0.5)

        return out

def model():

    enc = Encoder(conv_dim=64, n_res_blocks=2)
    dec = Decoder(conv_dim=64, n_res_blocks=2)
    return nn.Sequential(enc, dec)

Output:
Instead of returning only the latent feature (output of last layer) from encoder you can return the output of intermediate layers along with the latent feature, may be as a list. Afterwards, in the decoder's forward function you can access the list of values returned from encoder (which is the parameters of decoder) and use that correspondingly in the decoder layer.

Hope this bit helps.


Instruction: torch stack producing wrong dimensions

Input:
I am quite new to pytorch, and I am trying to create data within a dataloader and my code looks like so:

a=[]
.. within a for loop
self.a.append(torch.stack([b[ith_idx][j], \
b[ith_idx][rnd_dist], \
b[rnd_cls_idx][rnd_dist_rnd_cls]]\
))
self.c.append([1,0])

where, b is a python list of tensors. For example, the first element of b has shape torch.Size([46, 3, 512, 512]) (3 channels 512 x 512).

        self.a = torch.stack(self.a)
        self.c = torch.tensor(self.c)

and I notice I have shapes of [500,3,3,512,512] and [500,2] for a and b, while I was expecting 500,3,3,512,512 and 500 as tensor shapes.

Any pointers as to why this is happening would be helpful.

Output:
The shape of c is [500, 2] because you are appending a list of two values, [1,0] each time to c inside the loop. If you append either 1 or 0 iteratively then you will have a list of 500 elements.

For instance, you can append 1 or 0 randomly like follows:

self.c.append(np.random.randint(2))  # import numpy as np

or

self.c.append(torch.randint(0,2,(1,)))


Instruction: masked_scatter but rowwise?

Input:
Assuming a mask as follows:

mask = torch.tensor([
  [True,  True,  False, True,  False],
  [True,  False, True,  True,  True ],
])

I would like to number the True values with sequential values in each row separately. I don't care what's in the False spots, so 0 for simplicity. Thus the desired result is

tensor([[0, 1, 0, 2, 0],    # 0 1 _ 2 _
        [0, 0, 1, 2, 3]])   # 0 _ 1 2 3

I hoped this would work:

replacements = torch.arange(mask.size(1)).expand(mask.size())
target = torch.zeros(mask.size(), dtype=int)
target.masked_scatter(mask, replacements)

Unfortunately, masked_scatter ignores the shape of replacements, so this code results in:

tensor([[0, 1, 0, 2, 0],    # 0 1 _ 2 _
        [3, 0, 4, 0, 1]])   # 3 _ 4 0 1

What would I need to do instead?

Output:
I would try something with torch.cumsum: torch.cumsum(mask,dim=1) -1) * mask

The complete example

import torch
mask = torch.tensor([
  [True,  True,  False, True,  False],
  [True,  False, True,  True,  True ],
])
result=torch.cumsum(mask,dim=1) -1) * mask
print(result)

That would print:

tensor([[0, 1, 0, 2, 0],
        [0, 0, 1, 2, 3]])


Instruction: Best practices to benchmark deep models on CPU (and not GPU) in PyTorch?

Input:
I am little uncertain about how to measure execution time of deep models on CPU in PyTorch ONLY FOR INFERENCE. I list here some of them but they maybe inaccurate. Please correct them if required and mention more if required. I am running on PyTorch version 1.3.1 and Intel Xeon with 64GB RAM, 3.5GHz processor and 8 cores.

Should we use time.time()?

I know that for GPU this is a very bad idea. For GPU I do as follows

with torch.no_grad():
    wTime = 0
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    out = model(input) # JUST FOR WARMUP

    start.record()
    for i in range(200):
        input = torch.rand(1,1,2048,2048).to(device)

#        beg = time.time() DO NOT USE FOR GPU

        got = net_amplifier(low,for_amplifier)

#        wTime+=time.time()-beg DO NOT USE FOR GPU
    end.record()
    torch.cuda.synchronize()

    print('execution time in MILLISECONDS: {}'.format(start.elapsed_time(end)/200))

For this code execution was done on GPU. If I have to run them on CPU what changes should be made? Will time.time() do?

Should we use volatile?

I think the use if volatile is now discouraged after v0.3. But will it still help if I use the eval mode and no_grad()

input = Variable(torch.randn(1, 3, 227, 227), volatile=True)
 model(input)

Should the page cache be cleared?

One way of doing this that I know is using sudo sh -c "/bin/echo 1 > /proc/sys/vm/drop_caches"

Should I remove nn.Sequential() and directly put in forward part

According to this link

  All the methods using copy_ take some time to execute, especially on CPU this might be slow. Also the nn.Sequential() modules are slower than just executing them on the forward pass. I think this is due to some overhead that needs to be created when executing the Sequential module.

Another thing which i do not understand on the same link is

  If you are running into performance issues with these small numbers, you might try to use torch.set_flush_denormal(True) to disable denormal floating point numbers on the CPU.

Should torch.set_num_threads(int) be used? If yes can a demo code be provided?
What does These context managers are thread local, so they won’t work if you send work to another thread using the :module:`threading` module, etc. mean as given in the documentation.

Please list any more issues for calculating execution time in CPU.
Thankyou

Output:
  Should we use time.time()?

Yes, it's fine for CPU

  Should we use volatile?

As you said it's deprecated. Since 0.4.0 torch.Tensor was merged with torch.Variable (it's deprecated as well) and torch.no_grad context manager should be used.

  Should the page cache be cleared?

I don't think so unless you know it's a problem

  Should I remove nn.Sequential() and directly put in forward part

No, torch.nn.Sequential should have no or negligible performance burden on your model. It's forward is only:

def forward(self, input):
    for module in self:
        input = module(input)
    return input

  If you are running into performance issues with these small numbers,
  you might try to use torch.set_flush_denormal(True) to disable
  denormal floating point numbers on the CPU.

Flushing denormal numbers (numbers which underflow) means replacing them strictly by 0.0 which might help with your performance if you have a lot of really small numbers. Example given by PyTorch docs:

>>> torch.set_flush_denormal(True)
True
>>> torch.tensor([1e-323], dtype=torch.float64)
tensor([ 0.], dtype=torch.float64)
>>> torch.set_flush_denormal(False)
True
>>> torch.tensor([1e-323], dtype=torch.float64)
tensor(9.88131e-324 *
       [ 1.0000], dtype=torch.float64)

  Should torch.set_num_threads(int) be used? If yes can a demo code be
  provided?

According to this document it might help if you don't allocate too many threads (probably at most as many as cores in your CPU so you might try 8).

So this piece at the beginning of your code might help:

torch.set_num_threads(8)

You may want to check numbers out and see whether and how much each value helps.

  What does These context managers are thread local, so they won’t work
  if you send work to another thread using the :module:threading
  module, etc. mean as given in the documentation.

If you use module like torch.multiprocessing and run torch.multiprocessing.spawn (or a-like) and one of your processes won't get into the context manager block the gradient won't be turned off (in case of torch.no_grad). Also if you use Python's threading only the threads where the block was run into will have gradients turned off (or on, it depends).

This code will make it clear for you:

import threading

import torch

def myfunc(i, tensor):
    if i % 2 == 0:
        with torch.no_grad():
            z = tensor * 2
    else:
        z = tensor * 2
    print(i, z.requires_grad)

if __name__ == "__main__":
    tensor = torch.randn(5, requires_grad=True)
    with torch.no_grad():
        for i in range(10):
            t = threading.Thread(target=myfunc, args=(i, tensor))
            t.start()

Which outputs (order may vary):

0 False
1 True
2 False
3 True
4 False
6 False
5 True
7 True
8 False
9 True

Also notice that torch.no_grad() in __main__ has no effect on spawned threads (neither would torch.enable_grad).

  Please list any more issues for calculating execution time in CPU.

Converting to torchscript (see here) might help, building PyTorch from source targeted at your architecture and it's capabilities and tons of other things, this question is too wide.


Instruction: Accuracy/train and Loss/train graph by tensorboard

Input: i used tensorboard for my pytorch project and got this result for accuracy/train and loss/train but i dont understand what it means

Output:
Your loss does not decrease and your accuracy does not increase during training. Not significantly.

First thing to try is adjusting the learning rate:
- One possibility is that the learning rate is too small, and therefore the weight updates are tiny and insignificant. Try increasing the learning rate by factor of x10 or even x100.
- On the other hand, your loss/accuracy do seem to oscillate, which may suggest update steps are too large. Try decreasing the learning rate by x10 and see if this oscillation subsides.


Instruction: Getting an Error in Pytorch: IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Input:
I seem to be having a problem with my code. The error occurs at:

x, predicted = torch.max(net(value).data.squeeze(), 1)

I'm not sure what the issue is, and I've tried everything to fix. From my understanding, there seems to be a problem with the tensor dimension. I'm not sure what else to do. Can anyone give me any suggestions or solutions on how to fix this problem? Thank you in advance.

class Network(nn.Module): #Class for the neural network
def __init__(self):
    super(Network, self).__init__()
    self.layer1 = nn.Linear(6, 10) #First number in the number of inputs(784 since 28x28 is 784.) Second number indicates the number of inputs for the hidden layer(can be any number).
    self.hidden = nn.Softmax() #Activation Function
    self.layer2 = nn.Linear(10, 1) #First number is the hidden layer number(same as first layer), second number is the number of outputs.
    self.layer3 = nn.Sigmoid()

def forward(self, x): #Feed-forward part of the neural network. We will will feed the input through every layer of our network.
    y = self.layer1(x)
    y = self.hidden(y)
    y = self.layer2(y)
    y = self.layer3(y)
    return y #Returns the result

net = Network()
loss_function = nn.BCELoss()
optimizer = optim.SGD(net.parameters(), lr=0.01)

for x in range(1): #Amount of epochs over the dataset
for index, value in enumerate(new_train_loader):
    print(value)#This loop loops over every image in the dataset
    #actual = value[0]
    actual_value = value[5]
    #print(value.size())
    #print(net(value).size())
    print("ACtual", actual_value)
    net(value)
    loss = loss_function(net(value), actual_value.unsqueeze(0)) #Updating our loss function for every image
    #Backpropogation
    optimizer.zero_grad() #Sets gradients to zero.
    loss.backward() #Computes gradients
    optimizer.step() #Updates gradients
    print("Loop #: ", str(x+1), "Index #: ", str(index+1), "Loss: ", loss.item())

right = 0
total = 0
for value in new_test_loader:
actual_value = value[5]
#print(torch.max(net(value).data, 1))
print(net(value).shape)
x, predicted = torch.max(net(value).data.squeeze(), 1)
total += actual_value.size(0)
right += (predicted==actual_value).sum().item()
print("Accuracy: " + str((100*right/total)))

I should also mention that i'm using the latest versions.

Output:
You are calling .squeeze() on the model's output, which removes all singular dimensions (dimensions that have size 1). Your model's output has size [batch_size, 1], so
 .squeeze() removes the second dimension entirely, resulting in size [batch_size]. After, you're trying to take the maximum value across dimension 1, but the only dimension you have is the 0th dimension.

You don't need to take the maximum value in this case, since you have only one class as the output, and with the sigmoid at the end of your model you get values between [0, 1]. Since your are doing a binary classification that single class acts as two, namely either it's 0 or it's 1. So it can be seen as the probability that it is the class 1. Then you just need to set use a threshold of 0.5, meaning when the probability is over 0.5 it's class 1 and if the probability is under 0.5 it's the class 0. That's exactly what rounding does, therefore you can use torch.round.

output = net(value)
predicted = torch.round(output.squeeze())

On a side note, you are calling net(value) multiple times with the same value, and that means that its output is calculated multiple times as well, because it needs to go through the entire network again. That is unnecessary and you should just save the output in a variable. With this small network it isn't noticeable, but with larger networks that will take a lot of unnecessary time to recalculate the output.


Instruction: Getting Bad Images After Data Augmentation in PyTorch

Input:
I'm working on a nuclear segmentation problem where I'm trying to identify the positions of nuclei in images of stained tissues. The given training dataset has a picture of the stained tissue and a mask with the nuclei positions. Since the dataset was small, I wanted to try data augmentation in PyTorch, but after doing that, for some reason, when I output my mask image, it looks fine, but the corresponding tissue image is incorrect.

All my training images are in X_train with shape (128, 128, 3), the corresponding masks in Y_train with shape (128, 128, 1) and similarly the cross-validation images and masks in X_val and Y_val respectively.

Y_train and Y_val have dtype = np.bool, X_train and X_val have dtype = np.uint8.

Before data augmentation, I check my images like this:

fig, axis = plt.subplots(2, 2)
axis[0][0].imshow(X_train[0].astype(np.uint8))
axis[0][1].imshow(np.squeeze(Y_train[0]).astype(np.uint8))
axis[1][0].imshow(X_val[0].astype(np.uint8))
axis[1][1].imshow(np.squeeze(Y_val[0]).astype(np.uint8))

The output is as follows:
Before Data Augmentation

For the data augmentation, I define a custom class as follows:

Here I have imported torchvision.transforms.functional as TF and torchvision.transforms as transforms. images_np and masks_np are the inputs which are numpy arrays.

class Nuc_Seg(Dataset):
def __init__(self, images_np, masks_np):
    self.images_np = images_np
    self.masks_np = masks_np

def transform(self, image_np, mask_np):
    ToPILImage = transforms.ToPILImage()
    image = ToPILImage(image_np)
    mask = ToPILImage(mask_np.astype(np.int32))

    angle = random.uniform(-10, 10)
    width, height = image.size
    max_dx = 0.2 * width
    max_dy = 0.2 * height
    translations = (np.round(random.uniform(-max_dx, max_dx)), np.round(random.uniform(-max_dy, max_dy)))
    scale = random.uniform(0.8, 1.2)
    shear = random.uniform(-0.5, 0.5)
    image = TF.affine(image, angle = angle, translate = translations, scale = scale, shear = shear)
    mask = TF.affine(mask, angle = angle, translate = translations, scale = scale, shear = shear)

    image = TF.to_tensor(image)
    mask = TF.to_tensor(mask)
    return image, mask

def __len__(self):
    return len(self.images_np)

def __getitem__(self, idx):
    image_np = self.images_np[idx]
    mask_np = self.masks_np[idx]
    image, mask = self.transform(image_np, mask_np)

    return image, mask

This is followed by:

I have used from torch.utils.data import DataLoader

train_dataset = Nuc_Seg(X_train, Y_train)
train_loader = DataLoader(train_dataset, batch_size = 16, shuffle = True)
val_dataset = Nuc_Seg(X_val, Y_val)
val_loader = DataLoader(val_dataset, batch_size = 16, shuffle = True)

After this step, I try to check on my first set training image and mask using this:

%matplotlib inline

for ex_img, ex_mask in train_loader:

    img = ex_img[0]
    img = img.reshape(128, 128, 3)
    mask = ex_mask[0]
    mask = mask.reshape(128, 128)

    img = img.numpy()
    mask = mask.numpy()

    fig, (axis_1, axis_2) = plt.subplots(1, 2)
    axis_1.imshow(img.astype(np.uint8))
    axis_2.imshow(mask.astype(np.uint8))

    break

I get this as my output:
After Data Augmentation 1

When I change the line axis_1.imshow(img.astype(np.uint8)) to axis_1.imshow(img),

I get this image:
After Data Augmentation 2

The images of the mask are correct, but for some reason, the images of nuclei are wrong. With the .astype(np.uint8), the tissue image is completely black.

Without the .astype(np.uint8), the positions of the nuclei are correct, but the color scheme is all messed up (I expect images like those seen before data augmentation, either gray-ish or pink-ish), plus 9 copies of the same image in a grid are displayed for some reason. Can you please help me get the correct output of the tissue images?

Output:
You are converting the images to PyTorch tensors, and in PyTorch the images have size [C, H, W]. When you are visualising them, you are converting the tensors back to NumPy arrays, where images have size [H, W, C]. Therefore you are trying to rearrange the dimensions, but you are using torch.reshape, which doesn't swap the dimensions, but only partitions the data in a different way.

An example makes this clearer:

# Incrementing numbers with size 2 x 3 x 3
image = torch.arange(2 * 3 * 3).reshape(2, 3, 3)
# => tensor([[[ 0,  1,  2],
#             [ 3,  4,  5],
#             [ 6,  7,  8]],
#
#            [[ 9, 10, 11],
#             [12, 13, 14],
#             [15, 16, 17]]])

# Reshape keeps the same order of elements but for a different size
# The numbers are still incrementing from left to right
image.reshape(3, 3, 2)
# => tensor([[[ 0,  1],
#             [ 2,  3],
#             [ 4,  5]],
#
#            [[ 6,  7],
#             [ 8,  9],
#             [10, 11]],
#
#            [[12, 13],
#             [14, 15],
#             [16, 17]]])

To reorder the dimensions you can use permute:

# Dimensions are swapped
# Now the numbers increment from top to bottom
image.permute(1, 2, 0)
# => tensor([[[ 0,  9],
#             [ 1, 10],
#             [ 2, 11]],
#
#            [[ 3, 12],
#             [ 4, 13],
#             [ 5, 14]],
#
#            [[ 6, 15],
#             [ 7, 16],
#             [ 8, 17]]])

  With the .astype(np.uint8), the tissue image is completely black.

PyTorch images are represented as floats with values between [0, 1], but NumPy uses integer values between [0, 255]. Casting the float values to np.uint8 will result in only 0s and 1s, where everything that was not equal to 1, will be set to 0, therefore the whole image is black.

You need to multiply the values by 255 to bring them into range of [0, 255].

img = img.permute(1, 2, 0) * 255
img = img.numpy().astype(np.uint8)

This conversion is also automatically done when you are converting a tensor to a PIL image with transforms.ToPILImage (or with TF.to_pil_image if you prefer the functional version) and the PIL image can be converted directly to a NumPy array. With that you don't have to worry about the dimensions, value ranges or type and the code above can be replaced with:

img = np.array(TF.to_pil_image(img))


Instruction: Multiplying matrices of different sizes

Input:
I have a matrix full of zeros except for one zone, where I have values other than 0:

i=20
j=30
z = np.zeros((50,50))
while i < 30:
    while j < 40:
        z[j, i] = np.random.rand(1,1)
        j+=1

    j=30
    i+=1

matshow(z)
np.where(z == z.max())

Then I have another matrix, smaller than the first one. The distribution of values in this matrix follows a Gaussian curve:

import numpy as np
from scipy import signal

def gkern(kernlen=21, std=3):
    """Returns a 2D Gaussian kernel array."""
    gkern1d = signal.gaussian(kernlen, std=std).reshape(kernlen, 1)
    gkern2d = np.outer(gkern1d, gkern1d)
    return gkern2d
gkern(21, 3)

What I want is the following:

Identify where the maximum value of the 50x50 matrix is located. On this value, I want to center the maximum value of the smallest matrix, and then do the multiplication between the value of my small matrix and the values of the big matrices that are convered by the smaller matrix.

Output:
First use the original code to make the big matrix:

i=20
j=30
z = np.zeros((50,50))
while i < 30:
    while j < 40:
        z[j, i] = np.random.rand(1,1)
        j+=1
    j=30
    i+=1

Then figure out where zmax is:

[x,y] = np.where(z==z.max());

Now we know where the matrix should be centered. Get the shape of the g_kern matrix. This will tell us the size of the submatrix that we need to construct by taking the values of our larger matrix "covered" by the shape of the g_kern matrix, centered around the max of the larger matrix

[a,b] = np.shape(g_kern_matrix)

Now we center the submatrix of the larger matrix at the position of z_max. Its shape is determined by the second smaller matrix:

#x range of larger matrix slice
[x_min,x_max] = [int(x-a/2),int(x+a/2)]
#y range of larger matrix slice
[y_min,y_max] = [int(y-b/2),int(y+b/2)]
#construct the submatrix of the larger matrix
submatrix = z[x_min:x_max,y_min:y_max]
#can take a look at the submatrix to see that it seems correct
plt.matshow(submatrix)
plt.show()

And now we can do the multiplication

#now element wise multiply(hadamard product) this with the gaussian curve matrix
z[x_min:x_max,y_min:y_max] = submatrix*g_kern_matrix
plt.matshow(z)
plt.show()

Here I have assumed that when you ask to multiply the g_kern matrix and the submatrix, you mean element wise multiplication, as you cannot multiply two nonsquare matrices of the same shape.


Instruction: net.zero_grad() vs optim.zero_grad() pytorch

Input: Here they mention the need to include optim.zero_grad() when training to zero the parameter gradients. My question is: Could I do as well net.zero_grad() and would that have the same effect? Or is it necessary to do  optim.zero_grad(). Moreover, what happens if I do both? If I do none, then the gradients get accumulated, but what does that exactly mean? do they get added? In other words, what's the difference between doing optim.zero_grad() and net.zero_grad(). I am asking because here, line 115 they use net.zero_grad() and it is the first time I see that, that is an implementation of a reinforcement learning algorithm, where one has to be especially careful with the gradients because there are multiple networks and gradients, so I suppose there is a reason for them to do net.zero_grad() as opposed to optim.zero_grad().

Output:
net.zero_grad() sets the gradients of all its parameters (including parameters of submodules) to zero. If you call optim.zero_grad() that will do the same, but for all parameters that have been specified to be optimised. If you are using only net.parameters() in your optimiser, e.g. optim = Adam(net.parameters(), lr=1e-3), then both are equivalent, since they contain the exact same parameters.

You could have other parameters that are being optimised by the same optimiser, which are not part of net, in which case you would either have to manually set their gradients to zero and therefore keep track of all the parameters, or you can simply call optim.zero_grad() to ensure that all parameters that are being optimised, had their gradients set to zero.

  Moreover, what happens if I do both?

Nothing, the gradients would just be set to zero again, but since they were already zero, it makes absolutely no difference.

  If I do none, then the gradients get accumulated, but what does that exactly mean? do they get added?

Yes, they are being added to the existing gradients. In the backward pass the gradients in respect to every parameter are calculated, and then the gradient is added to the parameters' gradient (param.grad). That allows you to have multiple backward passes, that affect the same parameters, which would not be possible if the gradients were overwritten instead of being added.

For example, you could accumulate the gradients over multiple batches, if you need bigger batches for training stability but don't have enough memory to increase the batch size. This is trivial to achieve in PyTorch, which is essentially leaving off optim.zero_grad() and delaying optim.step() until you have gathered enough steps, as shown in HuggingFace - Training Neural Nets on Larger Batches: Practical Tips for 1-GPU, Multi-GPU & Distributed setups.

That flexibility comes at the cost of having to manually set the gradients to zero. Frankly, one line is a very small cost to pay, even though many users won't make use of it and especially beginners might find it confusing.


Instruction: PyTorch "Caught IndexError in DataLoader worker process 0", "IndexError: too many indices for array"

Input:
I am trying to implement a detection model based on "finetuning object detection" official tutorial of PyTorch.
It seemed to have worked with minimal data, (for 10 of images). However I uploaded my whole dataset to Drive and checked the index-data-label correspondences. There are not unmatching items in my setup, I have all the errors in that part solved. (I deleted extra items from the labels on GDrive)

class SomeDataset(torch.utils.data.Dataset):
def __init__(self, root_path, transforms):
    self.root_path = root_path
    self.transforms = transforms
    # load all image files, sorting them to
    # ensure that they are aligned
    self.imgs = list(sorted(os.listdir(os.path.join(root_path, "images"))))
    self.labels = list(sorted(os.listdir(os.path.join(root_path, "labels"))))

def __getitem__(self, idx):
    # load images ad masks
    img_path = os.path.join(self.root_path, "images", self.imgs[idx])
    label_path = os.path.join(self.root_path, "labels", self.labels[idx])

    img = Image.open(img_path).convert("RGB")

    # get labels and boxes
    label_data = np.loadtxt(label_path, dtype=str, delimiter=' ');
    print(f"{len(label_data)} is the length of label data")
    num_objs = label_data.shape[0];
    if num_objs != 0:
        print(f"number of objects {num_objs}")
        # label values should start from 1
        for i,label_name in enumerate(classnames):
            label_data[np.where(label_name==label_data)] = i;

        label_data = label_data.astype(np.float);
        print(f"label data {label_data}")
        xs = label_data[:,0:8:2];
        ys = label_data[:,1:8:2];

        x_min = np.min(xs, axis=1)[...,np.newaxis];
        x_max = np.max(xs, axis=1)[...,np.newaxis];
        y_min = np.min(ys, axis=1)[...,np.newaxis];
        y_max = np.max(ys, axis=1)[...,np.newaxis];

        boxes = np.hstack((x_min,y_min,x_max,y_max));

        labels = label_data[:,8];
    else:
        # if there is no label add background whose label is 0
        boxes = [[0,0,1,1]];
        labels = [0];
        num_objs = 1;

    boxes = torch.as_tensor(boxes, dtype=torch.float32)
    labels = torch.as_tensor(labels, dtype=torch.int64)

    image_id = torch.tensor([idx])
    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    # suppose all instances are not crowd
    iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

    target = {}
    target["boxes"] = boxes
    target["labels"] = labels
    target["image_id"] = image_id
    target["area"] = area
    target["iscrowd"] = iscrowd

    if self.transforms is not None:
          img, target = self.transforms(img, target)

    return img, target

def __len__(self):
    return len(self.imgs)

My main method is like the following,

def main():
# train on the GPU or on the CPU, if a GPU is not available
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

# our dataset has 16 classes - background and others
num_classes = 16
# use our dataset and defined transformations
dataset = SomeDataset('trainImages', get_transform(train=True))
print(f"{len(dataset)} number of images in training dataset")
dataset_validation = SomeDataset('valImages', get_transform(train=True))
print(f"{len(dataset_validation)} number of images in validation dataset")

# define training and validation data loaders
data_loader = torch.utils.data.DataLoader(
    dataset, batch_size=20, shuffle=True, num_workers=4,
    collate_fn=utils.collate_fn)

data_loader_val = torch.utils.data.DataLoader(
    dataset_validation, batch_size=10, shuffle=False, num_workers=4,
    collate_fn=utils.collate_fn)

# get the model using our helper function
#model = get_model_instance_segmentation(num_classes)
model = get_rcnn(num_classes);

# move model to the right device
model.to(device)

# construct an optimizer
params = [p for p in model.parameters() if p.requires_grad]
#optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005);
optimizer = torch.optim.Adam(params, lr=0.0005);
# and a learning rate scheduler
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=3,
                                               gamma=0.1)

# let's train it for 10 epochs
num_epochs = 5

for epoch in range(num_epochs):
    # train for one epoch, printing every 10 iterations
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=100)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    #evaluate(model, data_loader_test, device=device)

print("That's it!")
return model;

When I run my code, it runs for a few number of data (for example 10 of them) and then stops and gives out this error.

IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "<ipython-input-114-e0ccd94603fd>", line 31, in __getitem__
    xs = label_data[:,0:8:2];
IndexError: too many indices for array

The error goes from model = main() to > train_one_epoch() and goes on.

I do not understand why this is happening.

Also, this is an example from one instance of dataset,

(<PIL.Image.Image image mode=RGB size=1024x1024 at 0x7F46FC0A94A8>, {'boxes': tensor([[ 628.,    6.,  644.,   26.],
    [ 633.,   50.,  650.,   65.],
    [ 620.,   27.,  637.,   44.],
    [ 424.,  193.,  442.,  207.],
    [ 474.,  188.,  496.,  204.],
    [ 383.,  226.,  398.,  236.],
    [ 399.,  218.,  418.,  231.],
    [  42.,  189.,   63.,  203.],
    [ 106.,  159.,  129.,  169.],
    [ 273.,   17.,  287.,   34.],
    [ 225.,  961.,  234.,  980.],
    [ 220., 1004.,  230., 1024.]]), 'labels': tensor([5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]), 'image_id': tensor([0]), 'area': tensor([320., 255., 289., 252., 352., 150., 247., 294., 230., 238., 171., 200.]), 'iscrowd': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])})

Output:
When using np.loadtxt() method, make sure to add ndims = 2 as a parameter.
Because the number of objects parameter num_obj becomes 10 even if it has only 1 object in it.

It is because 1 object becomes a column vector which shows up as 10 objects. (representing 10 columns)

ndims = 2, makes sure that the output of np.loadtxt() method does not give out any row or column vectors, only 2 dimensional outputs.


Instruction: ResNet50 torchvision implementation gives low accuracy on CIFAR-10

Input:
I am new to Deep Learning and PyTorch. I am using the resnet-50 model in the torchvision module on cifar10. I have imported the CIFAR-10 dataset from torchvision. The accuracy is very low on testing and I have tried configuring the classification layers but there is no change in the accuracy. Is there something wrong with my code? Am I making a mistake in calculating the accuracy?

import torchvision
import torch
import torch.nn as nn
from torch import optim
import os
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
import numpy as np
from collections import OrderedDict
import matplotlib.pyplot as plt

transformations=transforms.Compose([transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])
trainset=torchvision.datasets.CIFAR10(root='./CIFAR10',download=True,transform=transformations,train=True)

testset=torchvision.datasets.CIFAR10(root='./CIFAR10',download=True,transform=transformations,train=False)

trainloader=DataLoader(dataset=trainset,batch_size=4)
testloader=DataLoader(dataset=testset,batch_size=4)

inputs,labels=next(iter(trainloader))
labels=labels.float()
inputs.size()

print(labels.type())
resnet=torchvision.models.resnet50(pretrained=True)

if torch.cuda.is_available():
  resnet=resnet.cuda()
  inputs,labels=inputs.cuda(),torch.Tensor(labels).cuda()

outputs=resnet(inputs)
outputs.size()

for param in resnet.parameters():
  param.requires_grad=False

numft=resnet.fc.in_features
print(numft)
resnet.fc=torch.nn.Sequential(nn.Linear(numft,1000),nn.ReLU(),nn.Linear(1000,10))
resnet.cuda()
resnet.train(True)
optimizer=torch.optim.SGD(resnet.parameters(),lr=0.001,momentum=0.9)
criterion=nn.CrossEntropyLoss()

for epoch in range(5):
    resnet.train(True)

    trainloss=0
    correct=0
    for x,y in trainloader:
        x,y=x.cuda(),y.cuda()
        optimizer.zero_grad()

        yhat=resnet(x)
        loss=criterion(yhat,y)

        loss.backward()
        optimizer.step()
        trainloss+=loss.item()

    print('Epoch: {} Loss: {}'.format(epoch,(trainloss/len(trainloader))))

    accuracy=[]
    running_corrects=0.0
    for x_test,y_test in testloader:

        x_test,y_test=x_test.cuda(),y_test.cuda()
        yhat=resnet(x_test)
        _,z=yhat.max(1)
        running_corrects += torch.sum(y_test == z)

    accuracy.append(running_corrects/len(testloader))

print(running_corrects/len(testloader))
accuracy=max(accuracy)
print(accuracy)

OUTPUT AFTER TRAINING/TESTING

Epoch: 0 Loss: 1.9808503997325897
Epoch: 1 Loss: 1.7917569598436356
Epoch: 2 Loss: 1.624434965057373
Epoch: 3 Loss: 1.4082191940283775
Epoch: 4 Loss: 1.1343850775527955
tensor(1.1404, device='cuda:0')
tensor(1.1404, device='cuda:0')

Output:
Couple of my observations:

You may want to fine-tune learning-rate and number of epochs and batch size. For example, currently you are training your model for only five epochs which might not be sufficient to achieve high accuracy. you can try with lager value of epochs.
Have you tried adapting backbone (feature extractor) model for CIFAR10 dataset by setting `param.requires_grad=True? Because the original model is trained on imagenet that might need to adapt on CIFAR10.
Before evaluation/testing you may like to set resnet.train(False) or resnet.eval() to let the model know that you are in eval mode. Furthermore, you may want to evaluate your model under the scope of no_grad() by using with torch.no_grad(): that will speed up inference time and reduce memory usage.
[CIFAR-10 is a balanced dataset so it's an optional (EDA) task here.] Have you checked the class distribution of CIFAR10 in terms of whether it's an imbalanced dataset or not? If it's an imbalanced dataset you may want to employ weighted cross entropy for you loss calculation. There are other strategies to tackle class-imbalance like over-sampling or under-sampling.

Regarding test accuracy, You need to divide the total number of correct prediction by the total number of samples in the dataset, len(testloader.dataset) instead of len(testloader). If you want your accuracy in the range of [0,100], just multiply by 100. You can print test accuracy for each epoch to check how it's changing whereas you are currently showing the maximum accuracy.


Instruction: How to set target in cross entropy loss for pytorch multi-class problem

Input:
Problem Statement: I have an image and a pixel of the image can belong to only(either) one of Band5','Band6', 'Band7' (see below for details). Hence, I have a pytorch multi-class problem but I am unable to understand how to set the targets which needs to be in form [batch, w, h]

My dataloader return two values:

x = chips.loc[:, :, :, self.input_bands]
y = chips.loc[:, :, :, self.output_bands]
x = x.transpose('chip','channel','x','y')
y_ohe = y.transpose('chip','channel','x','y')

Also, I have defined:

input_bands = ['Band1','Band2', 'Band3', 'Band3', 'Band4']  # input classes
output_bands = ['Band5','Band6', 'Band7'] #target classes

model = ModelName(num_classes = 3, depth=default_depth, in_channels=5, merge_mode='concat').to(device)
loss_new = nn.CrossEntropyLoss()

In my training function:

        #get values from dataloader
        X = normalize_zero_to_one(X) #input
        y = normalize_zero_to_one(y) #target

        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device)
        optim.zero_grad()
        outputs = model(images)

        loss = loss_new(outputs, masks) # (preds, target)
        loss.backward()
        optim.step() # Update weights

I know the the target (here masks) should be [batch_size, w, h]. However, it is currently [batch_size, channels, w, h].

I read a lot of posts including 1, 2 and they say the target should only contain the target class indices. I don't understand how can I concatenate indices of three classes and still set target as [batch_size, w, h].

Right now, I get the error:

RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4

To the best of my understanding, I don't need to do any one hot encoding. Similar errors and explanation I found on the internet are here:'

Reference 1
Reference 2
Reference 3
Reference 4

Any help will be appreciated! Thank you.

Output:
If I understand correctly, your current "target" is [batch_size, channels, w, h] with channels==3 as you have three possible targets.
What are the values in your target represent? You basically have a 3-vector target for each pixel - are these the expected class probabilities? Are they "one-hot-vectors" indicating the correct "band"?
If so, you can get the target indices by simply taking the argmax along the target channel dimension:

proper_target = torch.argmax(masks, dim=1)  # make sure keepdim=False
loss = loss_new(outputs, proper_target)


Instruction: Impact of data shuffling on results reproducibility in Pytorch

Input:
Pytorch dataloader class has the following constructor:

DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,
           batch_sampler=None, num_workers=0, collate_fn=None,
           pin_memory=False, drop_last=False, timeout=0,
           worker_init_fn=None)

When shuffle is set to True, data is reshuffled at every epoch. Shuffling the order in which examples are fed to the classifier is helpful so that batches between epochs do not look alike. Doing so will eventually make our model more robust.

However, I'm unable to understand by setting shuffle=True, can we get the same accuracy value for different runs?

Output:
The main algorithm/principal deep learning is based on is weight optimization using stochastic gradient descend (and its variants). Being a stochastic algorithm you cannot expect to get exactly the same results if you run your algorithm multiple times.
In fact, you should see some variations, but they should be "roughly the same".

If you need to have exactly the same results when running your algorithm multiple times, you should look into reproducibility of results - which is a very delicate subject.

In summary:
1. If you do not shuffle at all, you will have perfect reproducibility, but the resulting accuracy are expected to be very low.
2. If you randomly shuffle (what most of the world does) you should expect slightly different accuracy value for each run, but they should all be significantly larger than the values of (1) "no shuffle".
3. If you follow the guidelines of reproducible results, you should have the exact same accuracy values for each run and they should be close to the values of (2) "shuffle".


Instruction: torch.nn.DataParallel and to(device) does not support nested modules

Input:
I have a torch.nn.module class defined in the following way:

class MyModule(torch.nn.Module):
    def __init__(self):
       super(MyModule, self).__init__()
       self.sub_module_a = ....  # nn.module
       self.sub_module_b_dict = {
          'B': .... # nn.module
       }

However after I call torch.nn.DataParallel(MyModule) and MyModule.to(device) only sub_module_a is put on cuda. The 'B' inside self.sub_module_b_dict is still on CPU.

Looks like DataParallel and to(device) only support first level variables inside a torch.nn.Module class. The modules nested inside a customized structure (in this case, a dictionary) seem to be ignored.

Am I missing some caveats here?

Output:
You MUST use proper nn containers for all nn.Module's methods to act recursively on sub modules.

In your case, 'B' module is stored in a simple pythonic dictionary. Replace this with [nn.ModuleDict] and you should be fine:

self.sub_module_b_dict = nn.ModuleDict({'B': ...})

See related threads:
a, b, c, d, e, to name just a few...


Instruction: Why can't I get this Runge-Kutta solver to converge as the time step decreases?

Input:
For reasons, I need to implement the Runge-Kutta4 method in PyTorch (so no, I'm not going to use scipy.odeint). I tried and I get weird results on the simplest test case, solving x'=x with x(0)=1 (analytical solution: x=exp(t)). Basically, as I reduce the time step, I cannot get the numerical error to go down. I'm able to do it with a simpler Euler method, but not with the Runge-Kutta 4 method, which makes me suspect some floating point issue here (maybe I'm missing some hidden conversion from double precision to single)?

import torch
import numpy as np
import matplotlib.pyplot as plt

def Euler(f, IC, time_grid):
    y0 = torch.tensor([IC])
    time_grid = time_grid.to(y0[0])
    values = y0

    for i in range(0, time_grid.shape[0] - 1):
        t_i = time_grid[i]
        t_next = time_grid[i+1]
        y_i = values[i]
        dt = t_next - t_i
        dy = f(t_i, y_i) * dt
        y_next = y_i + dy
        y_next = y_next.unsqueeze(0)
        values = torch.cat((values, y_next), dim=0)

    return values

def RungeKutta4(f, IC, time_grid):

    y0 = torch.tensor([IC])
    time_grid = time_grid.to(y0[0])
    values = y0

    for i in range(0, time_grid.shape[0] - 1):
        t_i = time_grid[i]
        t_next = time_grid[i+1]
        y_i = values[i]
        dt = t_next - t_i
        dtd2 = 0.5 * dt
        f1 = f(t_i, y_i)
        f2 = f(t_i + dtd2, y_i + dtd2 * f1)
        f3 = f(t_i + dtd2, y_i + dtd2 * f2)
        f4 = f(t_next, y_i + dt * f3)
        dy = 1/6 * dt * (f1 + 2 * (f2 + f3) +f4)
        y_next = y_i + dy
        y_next = y_next.unsqueeze(0)
        values = torch.cat((values, y_next), dim=0)

    return values

# differential equation
def f(T, X):
    return X

# initial condition
IC = 1.

# integration interval
def integration_interval(steps, ND=1):
    return torch.linspace(0, ND, steps)

# analytical solution
def analytical_solution(t_range):
    return np.exp(t_range)

# test a numerical method
def test_method(method, t_range, analytical_solution):
    numerical_solution = method(f, IC, t_range)
    L_inf_err = torch.dist(numerical_solution, analytical_solution, float('inf'))
    return L_inf_err

if __name__ == '__main__':

    Euler_error = np.array([0.,0.,0.])
    RungeKutta4_error = np.array([0.,0.,0.])
    indices = np.arange(1, Euler_error.shape[0]+1)
    n_steps = np.power(10, indices)
    for i, n in np.ndenumerate(n_steps):
        t_range = integration_interval(steps=n)
        solution = analytical_solution(t_range)
        Euler_error[i] = test_method(Euler, t_range, solution).numpy()
        RungeKutta4_error[i] = test_method(RungeKutta4, t_range, solution).numpy()

    plots_path = "./plots"
    a = plt.figure()
    plt.xscale('log')
    plt.yscale('log')
    plt.plot(n_steps, Euler_error, label="Euler error", linestyle='-')
    plt.plot(n_steps, RungeKutta4_error, label="RungeKutta 4 error", linestyle='-.')
    plt.legend()
    plt.savefig(plots_path + "/errors.png")

The result:

As you can see, the Euler method converges (slowly, as expected of a first order method). However, the Runge-Kutta4 method does not converge as the time step gets smaller and smaller. The error goes down initially, and then up again. What's the issue here?

Output:
The reason is indeed a floating point precision issue. torch defaults to single precision, so once the truncation error becomes small enough, the total error is basically determined by the roundoff error, and reducing the truncation error further by increasing the number of steps <=> decreasing the time step doesn't lead to any decrease in the total error.

To fix this, we need to enforce double precision 64bit floats for all floating point torch tensors and numpy arrays. Note that the right way to do this is to use respectively torch.float64 and np.float64 rather than, e.g., torch.double and np.double, because the former are fixed-sized float values,  (always 64bit) while the latter depend on the machine and/or compiler. Here's the fixed code:

import torch
import numpy as np
import matplotlib.pyplot as plt

def Euler(f, IC, time_grid):

    y0 = torch.tensor([IC], dtype=torch.float64)
    time_grid = time_grid.to(y0[0])
    values = y0

    for i in range(0, time_grid.shape[0] - 1):
        t_i = time_grid[i]
        t_next = time_grid[i+1]
        y_i = values[i]
        dt = t_next - t_i
        dy = f(t_i, y_i) * dt
        y_next = y_i + dy
        y_next = y_next.unsqueeze(0)
        values = torch.cat((values, y_next), dim=0)

    return values

def RungeKutta4(f, IC, time_grid):

    y0 = torch.tensor([IC], dtype=torch.float64)
    time_grid = time_grid.to(y0[0])
    values = y0

    for i in range(0, time_grid.shape[0] - 1):
        t_i = time_grid[i]
        t_next = time_grid[i+1]
        y_i = values[i]
        dt = t_next - t_i
        dtd2 = 0.5 * dt
        f1 = f(t_i, y_i)
        f2 = f(t_i + dtd2, y_i + dtd2 * f1)
        f3 = f(t_i + dtd2, y_i + dtd2 * f2)
        f4 = f(t_next, y_i + dt * f3)
        dy = 1/6 * dt * (f1 + 2 * (f2 + f3) +f4)
        y_next = y_i + dy
        y_next = y_next.unsqueeze(0)
        values = torch.cat((values, y_next), dim=0)

    return values

    # differential equation
def f(T, X):
    return X

# initial condition
IC = 1.

# integration interval
def integration_interval(steps, ND=1):
    return torch.linspace(0, ND, steps, dtype=torch.float64)

# analytical solution
def analytical_solution(t_range):
    return np.exp(t_range, dtype=np.float64)

# test a numerical method
def test_method(method, t_range, analytical_solution):
    numerical_solution = method(f, IC, t_range)
    L_inf_err = torch.dist(numerical_solution, analytical_solution, float('inf'))
    return L_inf_err

if __name__ == '__main__':

    Euler_error = np.array([0.,0.,0.], dtype=np.float64)
    RungeKutta4_error = np.array([0.,0.,0.], dtype=np.float64)
    indices = np.arange(1, Euler_error.shape[0]+1)
    n_steps = np.power(10, indices)
    for i, n in np.ndenumerate(n_steps):
        t_range = integration_interval(steps=n)
        solution = analytical_solution(t_range)
        Euler_error[i] = test_method(Euler, t_range, solution).numpy()
        RungeKutta4_error[i] = test_method(RungeKutta4, t_range, solution).numpy()

    plots_path = "./plots"
    a = plt.figure()
    plt.xscale('log')
    plt.yscale('log')
    plt.plot(n_steps, Euler_error, label="Euler error", linestyle='-')
    plt.plot(n_steps, RungeKutta4_error, label="RungeKutta 4 error", linestyle='-.')
    plt.legend()
    plt.savefig(plots_path + "/errors.png")

Result:

Now, as we decrease the time step, the error of the RungeKutta4 approximation decreases with the correct rate.


Instruction: pytorch load _IncompatibleKeys

Input:
I trained a model of Efficentnet-b6(Architechture is as follows):

https://github.com/lukemelas/EfficientNet-PyTorch

Now, I tried to load a model I trained with it:

checkpoint  = torch.load('model.pth', map_location=torch.device('cpu'))
model.load_state_dict(checkpoint, strict=False)

but then I got the following error:

_IncompatibleKeys

missing_keys=['_conv_stem.weight', '_bn0.weight', '_bn0.bias', ...]
unexpected_keys=['module._conv_stem.weight', 'module._bn0.weight', 'module._bn0.bias', ...]

Please let me know how can I fix that, what am I missing?
Thank you!

Output:
If you compare the missing_keys and unexpected_keys, you may realize what is happening.

missing_keys=['_conv_stem.weight', '_bn0.weight', '_bn0.bias', ...]
unexpected_keys=['module._conv_stem.weight', 'module._bn0.weight', 'module._bn0.bias', ...]

As you can see, the model weights are saved with a module. prefix.
And this is because you have trained the model with DataParallel.

Now, to load the model weights without using DataParallel, you can do the following.

# original saved file with DataParallel
checkpoint = torch.load(path, map_location=torch.device('cpu'))

# create new OrderedDict that does not contain `module.`
from collections import OrderedDict

new_state_dict = OrderedDict()
for k, v in checkpoint.items():
    name = key.replace("module.", "") # remove `module.`
    new_state_dict[name] = v

# load params
model.load_state_dict(new_state_dict, strict=False)

OR, if you wrap the model using DataParallel, then you do not need the above approach.

checkpoint  = torch.load('model.pth', map_location=torch.device('cpu'))
model = torch.nn.DataParallel(model)
model.load_state_dict(checkpoint, strict=False)

Although the second approach is not encouraged (since you may not need DataParallel in many cases).


Instruction: Does Pytorch 1.4 support cuda10.1?

Input:
I try to install pytorch 1.5+cu101, but it can't find caffe2_nvrtc.dll(it indeed exists in lib\caffe2_nvrtc.dll). So I try to install pytorch 1.4+cu101, but I can not find such wheel(only pytorch1.4+cu100/cu92) from https://download.pytorch.org/whl/torch_stable.html.
Any help?

Output:
PyTorch 1.4.0 shipped with CUDA 10.1 by default, so there is no separate package with the cu101 suffix, those are only for alternative versions. You just need to install the regular torch package:

pip install torch==1.4.0 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: convert float image array to int in PIL via Image.fromarray

Input:
I have a PyTorch tensor of shape 3,256,256 where 3 is the number of channels and the image dimensions are 256, with all float values.

I am trying to feed this into the network and use PIL to do some transforms. To this end, I do:

img = Image.fromarray((255*imgs[i]).numpy().astype(np.uint8))

but I get:

TypeError: Cannot handle this data type: (1, 1, 256), |u1

When I check the output of (255*imgs[i]).numpy().astype(np.uint8), I do however see, for example:

[[[ 62  57  59 ...  63  46  36]
  [ 72  71  67 ...  80  76  82]
  [ 58  63  63 ... 145 152 169]
  ...
  [238 240 243 ...   7   7   7]
  [241 239 240 ...   5   5   6]
  [241 243 242 ...   4   3   5]]

 [[ 83  78  80 ...  86  70  61]
  [ 91  90  85 ...  95  93  98]
  [ 80  83  80 ... 141 150 168]
  ...
  [176 178 181 ...  14  14  14]
  [177 176 178 ...  15  15  17]
  [179 180 180 ...  13  13  15]]

 [[147 141 143 ... 150 136 128]
  [147 149 148 ... 154 149 154]
  [141 149 148 ... 178 182 196]
  ...
  [129 131 134 ...  43  43  43]
  [130 130 131 ...  45  45  47]
  [133 134 133 ...  44  44  46]]]

I am not an image expert by a long shot, and I am struggling to troubleshoot this issue now.

Output:
You need to match the order of dimensions, not only the dtype. PIL.Image expects its RGB images to be of shape hxwx3 - channel dimension last, while pytorch prefer to represent images as 3xhxw - channel dimension first.

Thus you need:

img = Image.fromarray((255*imgs[i]).numpy().astype(np.uint8).transpose(1, 2, 0))


Instruction: Pytorch isnt found by my code using import torch

Input:
I have installed torch using pip. And when I try to run

import torch

I get an error saying

raceback (most recent call last):
  File "C:\Users\leoga\Documents\AI Image Classifier\Classifier.py", line 2, in <module>
    import torch
  File "C:\Python38\lib\site-packages\torch\__init__.py", line 81, in <module>
    ctypes.CDLL(dll)
  File "C:\Python38\lib\ctypes\__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'C:\Python38\lib\site-packages\torch\lib\caffe2_nvrtc.dll'. Try using the full path with constructor syntax.

I am using python 3.8

Output:
Have you had python 2.x installed on your computer? It is possible that you need to use

pip3 install torch

if you use just pip instead of pip3 it could install the library in the wrong version of python. If you are on a mac then this is probably the case, as python 2 comes preinstalled on the machine.


Instruction: Is there a PyTorch tensor function that reduces dimension in regular patterns

Input:
Sorry for the poor wording of the title. What I wanted to do is like this:

Matrix 1 is the original matrix, and matrix 2 is matrix 1 but with every even columns and rows taken out. Matrix 3 is matrix 1 but only has 1 (mod 3) columns and rows. Matrix 4 is the same, with 1 (mod 4) columns and rows. Matrix 5 has 1 (mod 2) columns and all rows.

Is there a PyTorch function that manipulates tensors in this way that is fast and can utilize the GPU? This is sort of like MaxPool2d, however I just need the first value and not the max. If there aren't any functions like that, is there a way to do it manually but still fast?

Output:
Matrix 5 is the easiest to show, because you only need to slice along one dimension. But you can slice along both to get the other results.

matrix5 = matrix1[, ::2]

This notation takes every second column, starting at the zeroth.


Instruction: How to load your image dataset in python

Input:
I have a folder (on my windows desktop) containing the images I want to use to build my deep learning classifier. I also have one .csv file which has the image number (for example img_1035) and the corresponding class label. How do I load the dataset with the labels into python/jupyter notebooks?
This is the link to the dataset on kaggle (https://www.kaggle.com/debdoot/bdrw).

I would preferably like to use PyTorch to do this but any other ways would also be highly appreciated.

Output:
Luckily, PyTorch has a convenient "ImageFolder" class that you can extend to create your own dataset.

Here's an example of a dataset that uses ImageFolder:

class MyDataset(torchvision.datasets.ImageFolder):

def __init__(self, train_folder_path='.', transform=None, target_transform=None):
    super().__init__(train_folder_path, transform, target_transform)

# [ Some functions omitted ]

Then you load your set using PyTorch's "DataLoader".
Here's an example for a training set:

training_set = MyDataset(root_path, transform)
train_loader = torch.utils.data.DataLoader(training_set, batch_size=batch_size, shuffle=True)

Using the train loader you can get batches from your dataset. You can then use these batches to train / validate and so on:

batch = next(iter(train_loader))
images, labels = batch

Training is a rather involved process so I'm not entirely sure how deep you want to dive here. I hope this was a nudge in the right direction.


Instruction: Unexpected error when loading the model: problem in predictor - ModuleNotFoundError: No module named 'torchvision'

Input:
I've been trying to deploy my model to the AI platform for Prediction through the console on my vm instance, but I've gotten the error "(gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error:  "Failed to load model: Unexpected error when loading the model: problem in predictor - ModuleNotFoundError: No module named 'torchvision' (Error code: 0)"

I need to include both torch and torchvision. I followed the steps in this question Cannot deploy trained model to Google Cloud Ai-Platform with custom prediction routine: Model requires more memory than allowed, but I couldn't fetch the files pointed to by user gogasca. I tried downloading this .whl file from Pytorch website and uploading it to my cloud storage but got the same error that there is no module torchvision, even though this version is supposed to include both torch and torchvision. Also tried using Cloud AI compatible packages here, but they don't include torchvision.

I tried pointing to two separate .whl files for torch and torchvision in the --package-uris arguments, those point to files in my cloud storage, but then I got the error that the memory capacity was exceeded. This is strange, because collectively their size is around 130Mb. An example of my command that resulted in absence of torchvision looked like this:

gcloud beta ai-platform versions create version_1 \
  --model online_pred_1 \
  --runtime-version 1.15 \
  --python-version 3.7 \
  --origin gs://BUCKET/model-dir \
  --package-uris gs://BUCKET/staging-dir/my_package-0.1.tar.gz,gs://BUCKET/torchvision-dir/torch-1.4.0+cpu-cp37-cp37m-linux_x86_64.whl \
  --prediction-class predictor.MyPredictor

I've tried pointing to different combinations of .whl files that I obtained from different sources, but got either the no module error or not enough memory. I don't understand how the modules interact in this case and why the compiler thinks there is no such module. How can I resolve this? Or alternatively, how can I compile a package myself that include both torch and torchvision. Can you please give detailed answers because I'm not very familiar with package management and bash scripting.

Here's the code I used, torch_model.py:

from torch import nn

class EthnicityClassifier44(nn.Module):
    def __init__(self, num_classes=2):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=7, stride=1, padding=3)
        self.maxpool1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv22 = nn.Conv2d(32, 32, kernel_size=3, stride=1, padding=1)
        self.maxpool2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1)
        self.maxpool3 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv4 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1)
        self.maxpool4 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.relu = nn.ReLU(inplace=False)
        self.fc1 = nn.Linear(8*8*128, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(128, num_classes)

    def forward(self, x):
        x = self.relu(self.conv1(x))
        x = self.maxpool1(x)
        x = self.relu(self.conv22(x))
        x = self.maxpool2(x)
        x = self.maxpool3(self.relu(self.conv3(x)))
        x = self.maxpool4(self.relu(self.conv4(x)))
        x = self.relu(self.fc1(x.view(x.shape[0], -1)))
        x = self.relu(self.fc2(x))
        x = self.fc4(x)
        return x

This is predictor_py:

from facenet_pytorch import MTCNN, InceptionResnetV1, extract_face
import torch
import torchvision
from torchvision import transforms
from torch.nn import functional as F
from PIL import Image
from sklearn.externals import joblib
import numpy as np
import os
import torch_model

class MyPredictor(object):

    import torch
    import torchvision

    def __init__(self, model, preprocessor, device):
        """Stores artifacts for prediction. Only initialized via `from_path`.
        """
        self._resnet = model
        self._mtcnn_mult = preprocessor
        self._device = device
        self.get_std_tensor = transforms.Compose([
            np.float32,
            np.uint8,
            transforms.ToTensor(),
        ])
        self.tensor2pil = transforms.ToPILImage(mode='RGB')
        self.trans_resnet = transforms.Compose([
            transforms.Resize((100, 100)),
            np.float32,
            transforms.ToTensor()
        ])

    def predict(self, instances, **kwargs):

        pil_transform = transforms.Resize((512, 512))

        imarr = np.asarray(instances)
        pil_im = Image.fromarray(imarr)
        image = pil_im.convert('RGB')
        pil_im_512 = pil_transform(image)

        boxes, _ = self._mtcnn_mult(pil_im_512)
        box = boxes[0]

        face_tensor = extract_face(pil_im_512, box, margin=40)
        std_tensor = self.get_std_tensor(face_tensor.permute(1, 2, 0))
        cropped_pil_im = self.tensor2pil(std_tensor)

        face_tensor = self.trans_resnet(cropped_pil_im)
        face_tensor4d = face_tensor.unsqueeze(0)
        face_tensor4d = face_tensor4d.to(self._device)

        prediction = self._resnet(face_tensor4d)
        preds = F.softmax(prediction, dim=1).detach().numpy().reshape(-1)
        print('probability of (class1, class2) = ({:.4f}, {:.4f})'.format(preds[0], preds[1]))

        return preds.tolist()

    @classmethod
    def from_path(cls, model_dir):
        import torch
        import torchvision
        import torch_model

        model_path = os.path.join(model_dir, 'class44_M40RefinedExtra_bin_no_norm_7860.joblib')
        classifier = joblib.load(model_path)

        mtcnn_path = os.path.join(model_dir, 'mtcnn_mult.joblib')
        mtcnn_mult = joblib.load(mtcnn_path)

        device_path = os.path.join(model_dir, 'device_cpu.joblib')
        device = joblib.load(device_path)

        return cls(classifier, mtcnn_mult, device)

And setup.py:

from setuptools import setup

REQUIRED_PACKAGES = ['opencv-python-headless', 'facenet-pytorch']

setup(
 name="my_package",
 version="0.1",
 include_package_data=True,
 scripts=["predictor.py", "torch_model.py"],
 install_requires=REQUIRED_PACKAGES
)

Output:
The solution was to place the following packages in thsetup.py file for the custom prediction code:

REQUIRED_PACKAGES = ['torchvision==0.5.0', 'torch @ https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl', 'opencv-python', 'facenet-pytorch']

I then had a different problem with custom class instantiation, but this article explains it well. So I was able to successfully deploy my model to the AI Platform for prediction.


Instruction: Where is the PatchGAN implementation in the official PyTorch CycleGAN repo?

Input:
There's mention in the paper and in the code comments of a 70x70 PatchGAN, although I can't find anywhere in the code this is explicitly implemented.

Does anyone know how it is implemented?

Output: I've figured it out. After the last conv layer of the PatchGAN (before average pool) the receptive field size is 70. So each neuron on the single channel feature map (which is 30x30) coming out of that conv layer has information from a 70x70 patch of the input. The corresponding patches overlap one another on the input.


Instruction: Two questions on DCGAN: data normalization and fake/real batch

Input:
I am analyzing a meta-learning class that uses DCGAN + Reptile within the image generation.

I have two questions about this code.

First question: why during DCGAN training (line 74)

training_batch = torch.cat ([real_batch, fake_batch])

is a training_batch made up of real examples (real_batch) and fake examples (fake_batch) created? Why is training done by mixing real and false images? I have seen many DCGANs, but never with training done in this way.

The second question: why is the normalize_data function (line 49) and the unnormalize_data function (line 55) used during training?

def normalize_data(data):
    data *= 2
    data -= 1
    return data

def unnormalize_data(data):
    data += 1
    data /= 2
    return data

The project uses the Mnist dataset, if I wanted to use a color dataset like CIFAR10, do I have to modify those normalizations?

Output:
Training GANs involves giving the discriminator real and fake examples. Usually, you will see that they are given in two separate occasions. By default torch.cat concatenates the tensors on the first dimension (dim=0), which is the batch dimensions. Therefore it just doubled the batch size, where the first half are the real images and the second half the fake images.

To calculate the loss, they adapt the targets, such that the first half (original batch size) is classified as real, and the second half is classified as fake. From initialize_gan:

self.discriminator_targets = torch.tensor([1] * self.batch_size + [-1] * self.batch_size, dtype=torch.float, device=device).view(-1, 1)

Images are represented with float values between [0, 1]. The normalisation changes that to produce values between [-1, 1]. GANs generally use tanh in the generator, therefore the fake images have values between [-1, 1], hence the real images should be in the same range, otherwise it would be trivial for the discriminator to distinguish the fake images from the real ones.

If you want to display these images, you need to unnormalise them first, i.e. convert them to values between [0, 1].

  The project uses the Mnist dataset, if I wanted to use a color dataset like CIFAR10, do I have to modify those normalizations?

No, you don't need to change them, because images in colour also have their values between [0, 1], there are simply more values, representing the 3 channels (RGB).


Instruction: Pad torch tensors of different sizes to be equal

Input:
I am looking for a way to take an image/target batch for segmentation and return the batch where the image dimensions have been changed to be equal for the whole batch.  I have tried this using the code below:

def collate_fn_padd(batch):
    '''
    Padds batch of variable length

    note: it converts things ToTensor manually here since the ToTensor transform
    assume it takes in images rather than arbitrary tensors.
    '''
    # separate the image and masks
    image_batch,mask_batch = zip(*batch)

    # pad the images and masks
    image_batch = torch.nn.utils.rnn.pad_sequence(image_batch, batch_first=True)
    mask_batch = torch.nn.utils.rnn.pad_sequence(mask_batch, batch_first=True)

    # rezip the batch
    batch = list(zip(image_batch, mask_batch))

    return batch

However, I get this error:

RuntimeError: The expanded size of the tensor (650) must match the existing size (439) at non-singleton dimension 2.  Target sizes: [3, 650, 650].  Tensor sizes: [3, 406, 439]

How do I efficiently pad the tensors to be of equal dimensions and avoid this issue?

Output:
rnn.pad_sequence only pads the sequence dimension, it requires all other dimensions to be equal. You cannot use it to pad images across two dimensions (height and width).

To pad an image torch.nn.functional.pad can be used, but you need to manually determine the height and width it needs to get padded to.

import torch.nn.functional as F

# Determine maximum height and width
# The mask's have the same height and width
# since they mask the image.
max_height = max([img.size(1) for img in image_batch])
max_width = max([img.size(2) for img in image_batch])

image_batch = [
    # The needed padding is the difference between the
    # max width/height and the image's actual width/height.
    F.pad(img, [0, max_width - img.size(2), 0, max_height - img.size(1)])
    for img in image_batch
]
mask_batch = [
    # Same as for the images, but there is no channel dimension
    # Therefore the mask's width is dimension 1 instead of 2
    F.pad(mask, [0, max_width - mask.size(1), 0, max_height - mask.size(0)])
    for mask in mask_batch
]

The padding lengths are specified in reverse order of the dimensions, where every dimension has two values, one for the padding at the beginning and one for the padding at the end. For an image with the dimensions [channels, height, width] the padding is given as: [width_beginning, width_end, height_beginning, height_top], which can be reworded to [left, right, top, bottom]. Therefore the code above pads the images to the right and bottom. The channels are left out, because they are not being padded, which also means that the same padding could be directly applied to the masks.


Instruction: PyTorch LSTM dimension

Input:
I am trying to teach my first LSTM with numeric sequence data to predict a singular value.

My training set is a numpy matrix shape=207x7 where each input vector has 7 features.
I am not sure how to setup and train my LSTM properly. I have CNN experience, but now first LSTM.

class LSTM_NN(nn.Module):
"""
Stores the network format
"""
def __init__(self, input_size=7, hidden_layer_size=100, output_size=1):
    super(self.__class__, self).__init__()
    self._hidden_layer_size = hidden_layer_size
    self._lstm = nn.LSTM(input_size, hidden_layer_size)
    self._hidden_cell = (torch.zeros(1, 1, self._hidden_layer_size),
                         torch.zeros(1, 1, self._hidden_layer_size))
    self._linear = nn.Linear(hidden_layer_size, output_size)

def forward(self, input_data):
    """
    Forward propagation
    """

    lstm_out, self._hidden_cell = self._lstm(
        torch.tensor(np.expand_dims(input_data, 0)),
        self._hidden_cell)

training_data.shape
# (200, 7)
model = LSTM_NN(input_size=training_data.shape[1])
model.forward(training_data)

But I get this error:

Expected hidden[0] size (1, 200, 100), got (1, 1, 100)
  File "train_lstm.py", line 44, in forward

Output:
Your input has size [batch_size, seq_len, num_features] = [1, 200, 7]. The LSTM on the other hand, expects the input to have size [seq_len, batch_size, num_features] (as described in nn.LSTM - Inputs).

You can either change the dimensions of your input, or you can set batch_first=True when creating the LSTM, if you prefer having batch size as the first dimension, in which case batch_size and seq_len are swapped and your current input size would be the expected one.

self._lstm = nn.LSTM(input_size, hidden_layer_size, batch_first=True)


Instruction: CPU version of "torch._C._nn.nll_loss" function

Input:
Is there a function for torch._C._nn.nll_loss that takes in a CPU input? I don't have enough GPU memory to run my function so I'm trying to run everything on CPU.
This is my specific error (look at the anaconda files)

Traceback (most recent call last):
  File "plot_parametric_pytorch.py", line 395, in <module>
    val_result = validate(val_loader, model, criterion, 0)
  File "plot_parametric_pytorch.py", line 228, in validate
    training=False, optimizer=None)
  File "plot_parametric_pytorch.py", line 169, in forward
    loss = criterion(output, target_var)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 932, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_nll_loss_forward

Output:
nll_loss works for both CPU and GPU, but the input and the target need to be on the same device. Yours are on different devices, where the first one (output) is on the CPU, but the second (target_var) is on the GPU.

You need to put target_var onto the CPU.

loss = criterion(output, target_var.cpu())


Instruction: Is the forward definition of a model executed sequentially in PyTorch or in parallel?

Input:
I wanted to know if instructions in forward definition of deep models class are executed sequentially? For example:

class Net(nn.Module):
    ...
    def forward(self,x):

        #### Group 1
        y = self.conv1(x)
        y = self.conv2(y)
        y = self.conv3(y)

        ### Group 2
        z = self.conv4(x)
        z = self.conv5(z)
        z = self.conv6(z)

        out = torch.cat((y,z),dim=1)
        return out

In this case Group1 and Group2 instructions can be parallelized. But will the forward definition understand this automatically or will they be executed sequentially? If no, then how to run them in parallel?

I am running PyTorch 1.3.1

Thankyou very much

Output:
They are executed sequentially, only the calculations of the operations are parallelised. As far as I'm aware, there is no direct way to let them run in parallel by PyTorch.

I'm assuming that you are expecting a performance improvement from running them in parallel, but that would be at best minimal and at worst a lot slower, because operations like convolutions are already heavily parallelised and unless the input is extremely small, all cores will be used permanently. Running multiple convolutions in parallel would result in a lot of context switches, except if you would distribute the available cores evenly, but that wouldn't really make it any faster than doing them sequentially with all cores instead.

You can observe the same behaviour if you run two PyTorch programs at the same time, for example running the following, which has 3 relatively common convolutions and uses 224x224 images (like ImageNet), which is small compared to what other models (e.g. object detection) use:

import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding=1)

    def forward(self, input):
        out = self.conv1(input)
        out = self.conv2(out)
        out = self.conv3(out)
        return out

input = torch.randn((10, 3, 224, 224))
model = Model().eval()

# Running it 100 times just to create a microbenchmark
for i in range(100):
    out = model(input)

To obtain information about context switches, /usr/bin/time can be used (not the built in time).

/usr/bin/time -v python bench.py

Single run:

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:22.68
Involuntary context switches: 857

Running two instances at the same time:

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:43.69
Involuntary context switches: 456753

To clarify, each of the instances took about 43 seconds, that's not the accumulated time.