Instruction: How To downgrade Torch Version for Google Colab

Input: I would like to downgrade the Torch version used in my Google Colab notebooks. How could I do that?

Output:
Run from your cell:

!pip install torch==version

Where version could be, for example, 1.3.0 (default is 1.4.0). You may have to downgrade torchvision appropriately as well so you would go with:

!pip install torch==1.3.0 torchvision==0.4.1

On the other hand PyTorch provides backward compatibility between major versions so there should be no need for downgrading.

Remember you have to restart your Google Colab for changes to take effect


Instruction: Constant Training Loss and Validation Loss

Input:
I am running a RNN model with Pytorch library to do sentiment analysis on movie review, but somehow the training loss and validation loss remained constant throughout the training. I have looked up different online sources but still stuck.

Can someone please help and take a look at my code?

Some parameters are specified by the assignment:

embedding_dim = 64

n_layers = 1

n_hidden = 128

dropout = 0.5

batch_size = 32

My main code

txt_field = data.Field(tokenize=word_tokenize, lower=True, include_lengths=True, batch_first=True)
label_field = data.Field(sequential=False, use_vocab=False, batch_first=True)

train = data.TabularDataset(path=part2_filepath+"train_Copy.csv", format='csv',
                            fields=[('label', label_field), ('text', txt_field)], skip_header=True)
validation = data.TabularDataset(path=part2_filepath+"validation_Copy.csv", format='csv',
                            fields=[('label', label_field), ('text', txt_field)], skip_header=True)

txt_field.build_vocab(train, min_freq=5)
label_field.build_vocab(train, min_freq=2)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
train_iter, valid_iter, test_iter = data.BucketIterator.splits(
    (train, validation, test),
    batch_size=32,
    sort_key=lambda x: len(x.text),
    sort_within_batch=True,
    device=device)

n_vocab = len(txt_field.vocab)
embedding_dim = 64
n_hidden = 128
n_layers = 1
dropout = 0.5

model = Text_RNN(n_vocab, embedding_dim, n_hidden, n_layers, dropout)

optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
criterion = torch.nn.BCELoss().to(device)

N_EPOCHS = 15
best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):
    train_loss, train_acc = RNN_train(model, train_iter, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iter, criterion)

My Model

class Text_RNN(nn.Module):
    def __init__(self, n_vocab, embedding_dim, n_hidden, n_layers, dropout):
        super(Text_RNN, self).__init__()
        self.n_layers = n_layers
        self.n_hidden = n_hidden
        self.emb = nn.Embedding(n_vocab, embedding_dim)
        self.rnn = nn.RNN(
            input_size=embedding_dim,
            hidden_size=n_hidden,
            num_layers=n_layers,
            dropout=dropout,
            batch_first=True
        )
        self.sigmoid = nn.Sigmoid()
        self.linear = nn.Linear(n_hidden, 2)

    def forward(self, sent, sent_len):
        sent_emb = self.emb(sent)
        outputs, hidden = self.rnn(sent_emb)
        prob = self.sigmoid(self.linear(hidden.squeeze(0)))

        return prob

The training function

def RNN_train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.train()
    for batch in iterator:
        text, text_lengths = batch.text
        predictions = model(text, text_lengths)
        batch.label = batch.label.type(torch.FloatTensor).squeeze()
        predictions = torch.max(predictions.data, 1).indices.type(torch.FloatTensor)
        loss = criterion(predictions, batch.label)
        loss.requires_grad = True
        acc = binary_accuracy(predictions, batch.label)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

The output I run on 10 testing reviews + 5 validation reviews

Epoch [1/15]:   Train Loss: 15.351 | Train Acc: 44.44%  Val. Loss: 11.052 |  Val. Acc: 60.00%
Epoch [2/15]:   Train Loss: 15.351 | Train Acc: 44.44%  Val. Loss: 11.052 |  Val. Acc: 60.00%
Epoch [3/15]:   Train Loss: 15.351 | Train Acc: 44.44%  Val. Loss: 11.052 |  Val. Acc: 60.00%
Epoch [4/15]:   Train Loss: 15.351 | Train Acc: 44.44%  Val. Loss: 11.052 |  Val. Acc: 60.00%
...

Appreciate if someone can point me to the right direction, I believe is something with the training code, since for most parts I follow this article:
https://www.analyticsvidhya.com/blog/2020/01/first-text-classification-in-pytorch/

Output:
In your training loop you are using the indices from the max operation, which is not differentiable, so you cannot track gradients through it. Because it is not differentiable, everything afterwards does not track the gradients either. Calling
loss.backward() would fail.

# The indices of the max operation are not differentiable
predictions = torch.max(predictions.data, 1).indices.type(torch.FloatTensor)
loss = criterion(predictions, batch.label)
# Setting requires_grad to True to make .backward() work, although incorrectly.
loss.requires_grad = True

Presumably you wanted to fix that by setting requires_grad, but that does not do what you expect, because no gradients are propagated to your model, since the only thing in your computational graph would be the loss itself, and there is nowhere to go from there.

You used the indices to get either 0 or 1, since the output of your model is essentially two classes, and you wanted the one with the higher probability. For the Binary Cross Entropy loss, you only need one class that has a value between 0 and 1 (continuous), which you get by applying the sigmoid function.

So you need change the output channels of the final linear layer to 1:

self.linear = nn.Linear(n_hidden, 1)

and in your training loop you can remove the torch.max call and also the requires_grad.

# Squeeze the model's output to get rid of the single class dimension
predictions = model(text, text_lengths).squeeze()
batch.label = batch.label.type(torch.FloatTensor).squeeze()
loss = criterion(predictions, batch.label)
acc = binary_accuracy(predictions, batch.label)
optimizer.zero_grad()
loss.backward()

Since you have only 1 class at the end, an actual prediction would be either 0 or 1 (nothing in between), to achieve that you can simply use 0.5 as the threshold, so everything below is considered a 0 and everything above is considered a 1. If you are using the binary_accuracy function of the article you were following, that is done automatically for you. They do that by rounding it with torch.round.


Instruction: How to apply the histogram function in pytorch to a specific axis?

Input:
I would like to use the torch.histc function to different samples in my training batch.
Here is an example:

>>> tt2 = torch.from_numpy(np.array([[-0.2, 1, 0.21], [-0.1, 0.32, 0.2]]))
>>> tt3 = torch.from_numpy(np.array([[-0.8, 0.6, 0.1], [-0.6, 0.5, 0.4]]))
>>> t = torch.cat((tt2, tt3), 1)
>>> t
tensor([[-0.2000,  1.0000,  0.2100, -0.8000,  0.6000,  0.1000],
        [-0.1000,  0.3200,  0.2000, -0.6000,  0.5000,  0.4000]],
       dtype=torch.float64)
>>> torch.histc(t, bins=1, min=0, max=5)
tensor([8.], dtype=torch.float64)

However, I don't want to apply the histogram function for all the values in t, I rather expect something like this:

>>> torch.histc(torch.tensor([[-0.2000,  1.0000,  0.2100, -0.8000,  0.6000,  0.1000]]), bins=1, min=0, max=5)
tensor([4.])
>>> torch.histc(torch.tensor([[-0.1000,  0.3200,  0.2000, -0.6000,  0.5000,  0.4000]]), bins=1, min=0, max=5)
tensor([4.])
>>>

And, finally, I want to aggregate all the histograms in the same tensor: tensor([[4.], [4.]]).
Thanks in advance

Output:
The problem is solved with this function, but I'm not sure this is the most pythonic way to do it:

import numpy as np

def funct(semembs_as, semembs_bs):
    t = torch.cat((semembs_as, semembs_bs), 1)
    # make prediction a value between 0.0 and 5.0
    l = [torch.histc(ti, bins=1, min=0, max=5) for ti in t]
    y = [list(e) for e in l]
    return torch.from_numpy(np.array(y))

t1 = torch.from_numpy(np.array([[-0.2, 1, 0.21], [-0.1, 0.32, 0.2]]))
t2 = torch.from_numpy(np.array([[0.7, 0.0, -0.6], [-0.6, 0.5, 0.4]]))

x = funct(t1, t2)
x

  tensor([[4.],
          [4.]], dtype=torch.float64)

If you have better solutions, don't hesitate to comment, please.


Instruction: Converting keras code to pytorch code with Conv1D layer

Input:
I have some keras code that I need to convert to Pytorch. I've done some research but so far I am not able to reproduce the results I got from keras. I have spent many hours on this any tips or help is very appreciated.

Here is the keras code I am dealing with. The input shape is (None, 105, 768) where None is the batch size and I want to apply Conv1D to the input. The desire output in keras is (None, 105)

x = tf.keras.layers.Dropout(0.2)(input)
x = tf.keras.layers.Conv1D(1,1)(x)
x = tf.keras.layers.Flatten()(x)
x = tf.keras.layers.Activation('softmax')(x)

What I've tried, but worse in term of results:

    self.conv1d = nn.Conv1d(768, 1, 1)
    self.dropout = nn.Dropout(0.2)
    self.softmax = nn.Softmax()

def forward(self, input):

    x = self.dropout(input)
    x = x.view(x.shape[0],x.shape[2],x.shape[1])
    x = self.conv1d(x)
    x = torch.squeeze(x, 1)
    x = self.softmax(x)

Output:
The culprit is your attempt to swap the dimensions of the input around, since Keras and PyTorch have different conventions for the dimension order.

x = x.view(x.shape[0],x.shape[2],x.shape[1])

.view() does not swap the dimensions, but changes which part of the data is part of a given dimension. You can consider it as a 1D array, then you decide how many steps you take to cover the dimension. An example makes it much simpler to understand.

# Let's start with a 1D tensor
# That's how the underlying data looks in memory.
x = torch.arange(6)
# => tensor([0, 1, 2, 3, 4, 5])

# How the tensor looks when using Keras' convention (expected input)
keras_version = x.view(2, 3)
# => tensor([[0, 1, 2],
#            [3, 4, 5]])

# Vertical isn't swapped with horizontal, but the data is arranged differently
# The numbers are still incrementing from left to right
incorrect_pytorch_version = keras_version.view(3, 2)
# => tensor([[0, 1],
#            [2, 3],
#            [4, 5]])

To swap the dimensions you need to use torch.transpose.

correct_pytorch_version = keras_version.transpose(0, 1)
# => tensor([[0, 3],
#            [1, 4],
#            [2, 5]])


Instruction: Pytorch - Project each row of a tensor to the column space of another tensor

Input:
Currently, I have a tensor A, and a tensor U where U is an orthogonal matrix and is of full rank(so that its columns is a set of basis of U's column space, and all columns, say, u_i, have a norm of 1).

I am trying to compute the projection of each row of A onto the column space of U, using the formula from this post.

Which is, to compute Proj(A).
Is there any convenient functions or better operations to achieve this?
Thanks.

Output:
If you already have a unit norm in each column of projection matrix, just simply

torch.mm(A,U)

should be sufficient.


Instruction: Binary DenseNet 121 Classifier only predicting positive with probability >0.5

Input:
I borrowed code from this github repo for training of a DenseNet-121 [https://github.com/gaetandi/cheXpert/blob/master/cheXpert_final.ipynb][1]

The github code is for 14 class classification on the CheXpert chest X-ray dataset. I've revised it for binary classification.

# initialize and load the model
pathModel = "/ds2/images/model_ones_2epoch_densenet.tar"#"m-epoch0-07032019-213933.pth.tar"

I initialize the 14 class model so I can use the pretrained weights:

model = DenseNet121(nnClassCount).cuda()

model = torch.nn.DataParallel(model).cuda()
modelCheckpoint = torch.load(pathModel)
model.load_state_dict(modelCheckpoint['state_dict'])

And then convert to binary classification:

nnClassCount = 1
model.module.densenet121.classifier = nn.Sequential(
    nn.Linear(1024, nnClassCount),
    nn.Sigmoid()
).cuda()
model = torch.nn.DataParallel(model).cuda()

And then train via:

batch, losst, losse = CheXpertTrainer.train(model, dataLoaderTrain, dataLoaderVal, nnClassCount, 100, timestampLaunch, checkpoint = None, weight_path = weight_path)

My training data is laid out in a 2 column csv with column headers ('Path' and 'Class-Positive'), with path locations in the first column and 0 or 1 in the second column. I used oversampling when compiling the training list so paths in the csv are roughly a 50/50 split between 0's and 1's...shuffled.

I use livelossplot to monitor training/validation loss and accuracy. My loss plots look as expected but accuracy plots are flatlined around 0.5 (which makes sense given the 50/50 data if the net is saying its 100% positive or negative). I'm assuming I'm doing something wrong in how I'm doing predictions, but maybe something in the training is incorrect.

For predictions and probabilities I'm running:

 varOutput = model(varInput)
    _, preds = torch.max(varOutput, 1)
    print('varshape: ',varOutput.shape)
    probs = torch.sigmoid(varOutput)

*My issue: preds are all coming out as 0 and probs all above 0.5 *

Here is the initial code from github:

import os
import numpy as np
import time
import sys
import csv
import cv2
import matplotlib.pyplot as plt

import torch
import torch.nn as nn
import torch.backends.cudnn as cudnn
import torchvision
import torchvision.transforms as transforms
import torch.optim as optim
import torch.nn.functional as tfunc
from torch.utils.data import Dataset
from torch.utils.data.dataset import random_split
from torch.utils.data import DataLoader
from torch.optim.lr_scheduler import ReduceLROnPlateau
from PIL import Image
import torch.nn.functional as func

from sklearn.metrics.ranking import roc_auc_score
import sklearn.metrics as metrics
import random

use_gpu = torch.cuda.is_available()
# Paths to the files with training, and validation sets.
# Each file contains pairs (path to image, output vector)
pathFileTrain = '../CheXpert-v1.0-small/train.csv'
pathFileValid = '../CheXpert-v1.0-small/valid.csv'

# Neural network parameters:
nnIsTrained = False                 #pre-trained using ImageNet
nnClassCount = 14                   #dimension of the output

# Training settings: batch size, maximum number of epochs
trBatchSize = 64
trMaxEpoch = 3

# Parameters related to image transforms: size of the down-scaled image, cropped image
imgtransResize = (320, 320)
imgtransCrop = 224

# Class names
class_names = ['No Finding', 'Enlarged Cardiomediastinum', 'Cardiomegaly', 'Lung Opacity',
               'Lung Lesion', 'Edema', 'Consolidation', 'Pneumonia', 'Atelectasis', 'Pneumothorax',
               'Pleural Effusion', 'Pleural Other', 'Fracture', 'Support Devices']
class CheXpertDataSet(Dataset):
    def __init__(self, image_list_file, transform=None, policy="ones"):
        """
        image_list_file: path to the file containing images with corresponding labels.
        transform: optional transform to be applied on a sample.
        Upolicy: name the policy with regard to the uncertain labels
        """
        image_names = []
        labels = []

        with open(image_list_file, "r") as f:
            csvReader = csv.reader(f)
            next(csvReader, None)
            k=0
            for line in csvReader:
                k+=1
                image_name= line[0]
                label = line[5:]

                for i in range(14):
                    if label[i]:
                        a = float(label[i])
                        if a == 1:
                            label[i] = 1
                        elif a == -1:
                            if policy == "ones":
                                label[i] = 1
                            elif policy == "zeroes":
                                label[i] = 0
                            else:
                                label[i] = 0
                        else:
                            label[i] = 0
                    else:
                        label[i] = 0

                image_names.append('../' + image_name)
                labels.append(label)

        self.image_names = image_names
        self.labels = labels
        self.transform = transform

    def __getitem__(self, index):
        """Take the index of item and returns the image and its labels"""

        image_name = self.image_names[index]
        image = Image.open(image_name).convert('RGB')
        label = self.labels[index]
        if self.transform is not None:
            image = self.transform(image)
        return image, torch.FloatTensor(label)

    def __len__(self):
        return len(self.image_names)
#TRANSFORM DATA

normalize = transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
transformList = []
#transformList.append(transforms.Resize(imgtransCrop))
transformList.append(transforms.RandomResizedCrop(imgtransCrop))
transformList.append(transforms.RandomHorizontalFlip())
transformList.append(transforms.ToTensor())
transformList.append(normalize)
transformSequence=transforms.Compose(transformList)
#LOAD DATASET

dataset = CheXpertDataSet(pathFileTrain ,transformSequence, policy="ones")
datasetTest, datasetTrain = random_split(dataset, [500, len(dataset) - 500])
datasetValid = CheXpertDataSet(pathFileValid, transformSequence)
#Problèmes de l'overlapping de patients et du transform identique ?

dataLoaderTrain = DataLoader(dataset=datasetTrain, batch_size=trBatchSize, shuffle=True,  num_workers=24, pin_memory=True)
dataLoaderVal = DataLoader(dataset=datasetValid, batch_size=trBatchSize, shuffle=False, num_workers=24, pin_memory=True)
dataLoaderTest = DataLoader(dataset=datasetTest, num_workers=24, pin_memory=True)
class CheXpertTrainer():

    def train (model, dataLoaderTrain, dataLoaderVal, nnClassCount, trMaxEpoch, launchTimestamp, checkpoint):

        #SETTINGS: OPTIMIZER & SCHEDULER
        optimizer = optim.Adam (model.parameters(), lr=0.0001, betas=(0.9, 0.999), eps=1e-08, weight_decay=1e-5)

        #SETTINGS: LOSS
        loss = torch.nn.BCELoss(size_average = True)

        #LOAD CHECKPOINT
        if checkpoint != None and use_gpu:
            modelCheckpoint = torch.load(checkpoint)
            model.load_state_dict(modelCheckpoint['state_dict'])
            optimizer.load_state_dict(modelCheckpoint['optimizer'])

        #TRAIN THE NETWORK
        lossMIN = 100000

        for epochID in range(0, trMaxEpoch):

            timestampTime = time.strftime("%H%M%S")
            timestampDate = time.strftime("%d%m%Y")
            timestampSTART = timestampDate + '-' + timestampTime

            batchs, losst, losse = CheXpertTrainer.epochTrain(model, dataLoaderTrain, optimizer, trMaxEpoch, nnClassCount, loss)
            lossVal = CheXpertTrainer.epochVal(model, dataLoaderVal, optimizer, trMaxEpoch, nnClassCount, loss)

            timestampTime = time.strftime("%H%M%S")
            timestampDate = time.strftime("%d%m%Y")
            timestampEND = timestampDate + '-' + timestampTime

            if lossVal < lossMIN:
                lossMIN = lossVal
                torch.save({'epoch': epochID + 1, 'state_dict': model.state_dict(), 'best_loss': lossMIN, 'optimizer' : optimizer.state_dict()}, 'm-epoch'+str(epochID)+'-' + launchTimestamp + '.pth.tar')
                print ('Epoch [' + str(epochID + 1) + '] [save] [' + timestampEND + '] loss= ' + str(lossVal))
            else:
                print ('Epoch [' + str(epochID + 1) + '] [----] [' + timestampEND + '] loss= ' + str(lossVal))

        return batchs, losst, losse
    #--------------------------------------------------------------------------------

    def epochTrain(model, dataLoader, optimizer, epochMax, classCount, loss):

        batch = []
        losstrain = []
        losseval = []

        model.train()

        for batchID, (varInput, target) in enumerate(dataLoaderTrain):

            varTarget = target.cuda(non_blocking = True)

            #varTarget = target.cuda()

            varOutput = model(varInput)
            lossvalue = loss(varOutput, varTarget)

            optimizer.zero_grad()
            lossvalue.backward()
            optimizer.step()

            l = lossvalue.item()
            losstrain.append(l)

            if batchID%35==0:
                print(batchID//35, "% batches computed")
                #Fill three arrays to see the evolution of the loss

                batch.append(batchID)

                le = CheXpertTrainer.epochVal(model, dataLoaderVal, optimizer, trMaxEpoch, nnClassCount, loss).item()
                losseval.append(le)

                print(batchID)
                print(l)
                print(le)

        return batch, losstrain, losseval

    #--------------------------------------------------------------------------------

    def epochVal(model, dataLoader, optimizer, epochMax, classCount, loss):

        model.eval()

        lossVal = 0
        lossValNorm = 0

        with torch.no_grad():
            for i, (varInput, target) in enumerate(dataLoaderVal):

                target = target.cuda(non_blocking = True)
                varOutput = model(varInput)

                losstensor = loss(varOutput, target)
                lossVal += losstensor
                lossValNorm += 1

        outLoss = lossVal / lossValNorm
        return outLoss

    #--------------------------------------------------------------------------------

    #---- Computes area under ROC curve
    #---- dataGT - ground truth data
    #---- dataPRED - predicted data
    #---- classCount - number of classes

    def computeAUROC (dataGT, dataPRED, classCount):

        outAUROC = []

        datanpGT = dataGT.cpu().numpy()
        datanpPRED = dataPRED.cpu().numpy()

        for i in range(classCount):
            try:
                outAUROC.append(roc_auc_score(datanpGT[:, i], datanpPRED[:, i]))
            except ValueError:
                pass
        return outAUROC

    #--------------------------------------------------------------------------------

    def test(model, dataLoaderTest, nnClassCount, checkpoint, class_names):

        cudnn.benchmark = True

        if checkpoint != None and use_gpu:
            modelCheckpoint = torch.load(checkpoint)
            model.load_state_dict(modelCheckpoint['state_dict'])

        if use_gpu:
            outGT = torch.FloatTensor().cuda()
            outPRED = torch.FloatTensor().cuda()
        else:
            outGT = torch.FloatTensor()
            outPRED = torch.FloatTensor()

        model.eval()

        with torch.no_grad():
            for i, (input, target) in enumerate(dataLoaderTest):

                target = target.cuda()
                outGT = torch.cat((outGT, target), 0).cuda()

                bs, c, h, w = input.size()
                varInput = input.view(-1, c, h, w)

                out = model(varInput)
                outPRED = torch.cat((outPRED, out), 0)
        aurocIndividual = CheXpertTrainer.computeAUROC(outGT, outPRED, nnClassCount)
        aurocMean = np.array(aurocIndividual).mean()

        print ('AUROC mean ', aurocMean)

        for i in range (0, len(aurocIndividual)):
            print (class_names[i], ' ', aurocIndividual[i])

        return outGT, outPRED
class DenseNet121(nn.Module):
    """Model modified.
    The architecture of our model is the same as standard DenseNet121
    except the classifier layer which has an additional sigmoid function.
    """
    def __init__(self, out_size):
        super(DenseNet121, self).__init__()
        self.densenet121 = torchvision.models.densenet121(pretrained=True)
        num_ftrs = self.densenet121.classifier.in_features
        self.densenet121.classifier = nn.Sequential(
            nn.Linear(num_ftrs, out_size),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.densenet121(x)
        return x
# initialize and load the model
model = DenseNet121(nnClassCount).cuda()
model = torch.nn.DataParallel(model).cuda()
timestampTime = time.strftime("%H%M%S")
timestampDate = time.strftime("%d%m%Y")
timestampLaunch = timestampDate + '-' + timestampTime

batch, losst, losse = CheXpertTrainer.train(model, dataLoaderTrain, dataLoaderVal, nnClassCount, trMaxEpoch, timestampLaunch, checkpoint = None)
print("Model trained")

Output:
It looks like you have adapted the training correctly for the binary classification, but the prediction wasn't, as you are still trying it as if it were a multi-class prediction.

The output of your model (varOutput) has the size (batch_size, 1), since there is only one class. The maximum across that dimension will always be 0, since that is the only class available, there is no separate class for 1.

This single class represents both cases (0 and 1), so you can consider it is a the probability of it being positive (1). To get the distinct value of either 0 or 1, you simply use a threshold of 0.5, so everything below that receives the class 0 and above that 1. This can be easily done with torch.round.

But you also have another problem, you're applying the sigmoid function twice in a row, once in the classifier nn.Sigmoid() and then afterwards again torch.sigmoid(varOutput). That is problematic, because sigmoid(0) = 0.5, hence all your probabilities are over 0.5.

The output of your model are already the probabilities, the only thing left is to round them:

probs = model(varInput)
# The .squeeze(1) is to get rid of the singular class dimension
preds = torch.round(probs).squeeze(1)


Instruction: Is it a good idea to Multiply loss().item by batch_size to get the loss of a batch when batch size is not a factor of train_size?

Input:
Suppose we have problem where we have 100 images and a batch size of 15. We have 15 images in all of out batches except our last batch which contains 10 images.

Suppose we have network training as:

network = Network()
optimizer = optim.Adam(network.parameters(),lr=0.001)

for epoch in range(5):

    total_loss = 0

    train_loader = torch.utils.data.DataLoader(train_set,batch_size=15)

    for batch in train_loader:
        images,labels = batch

        pred = network(images)
        loss = F.cross_entropy(pred,labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        total_loss+= loss.item()*15

Is'nt the last batch always supposed to give us a increased value of loss because we will be multiplying by 15 where we were supposed to multiply by 10 in the last batch? Should't it be
total_loss+= loss.item()*len(images) in place of 15 or batch_size??

Can we use

for every epoch:
    for every batch:
        loss = F.cross_entropy(pred,labels,reduction='sum')
        total_loss+=loss.item()

    avg_loss_per_epoch = (total_loss/len(train_set))

can someone please explain that multiplying by batch_size a good idea and how am I wrong?

Output:
Yes, you're right. Usually, for running loss the term

total_loss+= loss.item()*15

is written instead as (as done in transfer learning tutorial)

total_loss+= loss.item()*images.size(0)

where images.size(0) gives the current batch size. Thus, it'll give 10 (in your case) instead of hard-coded 15 for the last batch. loss.item()*len(images) is also correct!

In your second example, since you're using reduction='sum', the loss won't be divided by the batch size as it's done by default (because, by default, the reduction='mean' i.e. losses are averaged across observations for each minibatch).


Instruction: How can I slice a PyTorch tensor with another tensor?

Input:
I have:

inp =  torch.randn(4, 1040, 161)

and I have another tensor called indices with values:

tensor([[124, 583, 158, 529],
        [172, 631, 206, 577]], device='cuda:0')

I want the equivalent of:

inp0 = inp[:,124:172,:]
inp1 = inp[:,583:631,:]
inp2 = inp[:,158:206,:]
inp3 = inp[:,529:577,:]

Except all added together, to have a .size of [4, 48, 161]. How can I accomplish this?

Currently, my solution is a for loop:

            left_indices = torch.empty(inp.size(0), self.side_length, inp.size(2))
            for batch_index in range(len(inp)):
                print(left_indices_start[batch_index].item())
                left_indices[batch_index] = inp[batch_index, left_indices_start[batch_index].item():left_indices_end[batch_index].item()]

Output:
Here you go (EDIT: you probably need to copy tensors to cpu using tensor=tensor.cpu() before doing following operations):

index = tensor([[124, 583, 158, 529],
    [172, 631, 206, 577]], device='cuda:0')
#create a concatenated list of ranges of indices you desire to slice
indexer = np.r_[tuple([np.s_[i:j] for (i,j) in zip(index[0,:],index[1,:])])]
#slice using numpy indexing
sliced_inp = inp[:, indexer, :]

Here is how it works:

np.s_[i:j] creates a slice object (simply a range) of indices from start=i to end=j.

np.r_[i:j, k:m] creates a list ALL indices in slices (i,j) and (k,m) (You can pass more slices to np.r_ to concatenate them all together at once. This is an example of concatenating only two slices.)

Therefore, indexer creates a list of ALL indices by concatenating a list of slices (each slice is a range of indices).

UPDATE: If you need to remove interval overlaps and sort intervals:

indexer = np.unique(indexer)

if you want to remove interval overlaps but not sort and keep original order (and first occurrences of overlaps)

uni = np.unique(indexer, return_index=True)[1]
indexer = [indexer[index] for index in sorted(uni)]


Instruction: Breaking down a batch in pytorch leads to different results, why?

Input:
I was trying something with batch processing in pytorch. In my code below, you may think of x as a batch of batch size 2 (each sample is a 10d vector). I use x_sep to denote the first sample in x.

import torch
import torch.nn as nn

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.fc1 = nn.Linear(10,10)

    def forward(self, x):
        x = self.fc1(x)
        return x

f = net()

x = torch.randn(2,10)
print(f(x[0])==f(x)[0])

Ideally, f(x[0])==f(x)[0] should give a tensor with all true entries. But the output on my computer is

tensor([False, False,  True,  True, False, False, False, False,  True, False])

Why does this happen? Is it a computational error? Or is it related to how the batch precessing in implemented in pytorch?

Update: I simplified the code a bit. The question remains the same.

My reasoning:
I believe f(x)[0]==f(x[0]) should have all its entries True because the law of matrix multiplication says so. Let us think of x as a 2x10 matrix, and think of the linear transformation f() as represented by matrix B (ignoring the bias for a moment). Then f(x)=xB by our notations. The law of matrix multiplication tells us that xB is equal to first multiply the two rows by B on the right separately, and then put the two rows back together. Translated back to the code, it is f(x[0])==f(x)[0] and f(x[1])==f(x)[1].

Even if we consider the bias, every row should have the same bias and the equality should still hold.

Also note that no training is done here. Hence how the weights are initialized shouldn't matter.

Output:
TL;DR
Under the hood it uses a function named addmm that have some optimizations, and probably multiply the vectors in a slightly different way

I just understood what was the real issue, and I edited the answer.
After trying to reproduce and debug it on my machine.
I found out that:
f(x)[0].detach().numpy()
>>>array([-0.5386441 ,  0.4983463 ,  0.07970242,  0.53507525,  0.71045876,
        0.7791027 ,  0.29027492, -0.07919329, -0.12045971, -0.9111403 ],
      dtype=float32)
f(x[0]).detach().numpy()
>>>array([-0.5386441 ,  0.49834624,  0.07970244,  0.53507525,  0.71045876,
        0.7791027 ,  0.29027495, -0.07919335, -0.12045971, -0.9111402 ],
      dtype=float32)
f(x[0]).detach().numpy() == f(x)[0].detach().numpy()
>>>array([ True, False, False,  True,  True,  True, False, False,  True,
   False])

If you give a close look, you will find out that all the indices which are False, there is a slight numeric change in 5th floating point.
After some more debugging, I saw in the linear function it uses addmm:
def linear(input, weight, bias=None):
    if input.dim() == 2 and bias is not None:
        # fused op is marginally faster
        ret = torch.addmm(bias, input, weight.t())
    else:
        output = input.matmul(weight.t())
    if bias is not None:
        output += bias
    ret = output
return ret

When addmm addmm,  implements beta*mat + alpha*(mat1 @ mat2) and is supposedly faster (see here for example).
Credit to Szymon Maszke


Instruction: Cannot find in-place operation causing "RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:"

Input:
I'm relatively new to PyTorch and am trying to reproduce an algorithm from an academic paper that approximates a term using the Hessian matrix.  I've set up a toy problem so that I can compare the results of the full Hessian with the approximation.  I found this gist and have been playing with it to compute the full Hessian part of the algorithm.

I am getting the error: "RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation."

I've scoured through the simple example code, documentation, and many, many forum posts about this issue and cannot find any in-place operations.  Any help would be greatly appreciated!

Here is my code:

import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np

torch.set_printoptions(precision=20, linewidth=180)

def jacobian(y, x, create_graph=False):
    jac = []
    flat_y = y.reshape(-1)
    grad_y = torch.zeros_like(flat_y)

    for i in range(len(flat_y)):
        grad_y[i] = 1.
        grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
        jac.append(grad_x.reshape(x.shape))
        grad_y[i] = 0.
    return torch.stack(jac).reshape(y.shape + x.shape)

def hessian(y, x):
    return jacobian(jacobian(y, x, create_graph=True), x)

def f(x):
    return x * x

np.random.seed(435537698)

num_dims = 2
num_samples = 3

X = [np.random.uniform(size=num_dims) for i in range(num_samples)]
print('X: \n{}\n\n'.format(X))

mean = torch.Tensor(np.mean(X, axis=0))
mean.requires_grad = True
print('mean: \n{}\n\n'.format(mean))

cov = torch.Tensor(np.cov(X, rowvar=False))
print('cov: \n{}\n\n'.format(cov))

with autograd.detect_anomaly():
    hessian_matrices = hessian(f(mean), mean)
    print('hessian: \n{}\n\n'.format(hessian_matrices))

And here is the output with the stack trace:

X:
[array([0.81700949, 0.17141617]), array([0.53579366, 0.31141496]), array([0.49756485, 0.97495776])]

mean:
tensor([0.61678934097290039062, 0.48592963814735412598], requires_grad=True)

cov:
tensor([[ 0.03043144382536411285, -0.05357056483626365662],
        [-0.05357056483626365662,  0.18426130712032318115]])

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-5a1c492d2873> in <module>()
     42
     43 with autograd.detect_anomaly():
---> 44     hessian_matrices = hessian(f(mean), mean)
     45     print('hessian: \n{}\n\n'.format(hessian_matrices))

2 frames
<ipython-input-3-5a1c492d2873> in hessian(y, x)
     21
     22 def hessian(y, x):
---> 23     return jacobian(jacobian(y, x, create_graph=True), x)
     24
     25 def f(x):

<ipython-input-3-5a1c492d2873> in jacobian(y, x, create_graph)
     15     for i in range(len(flat_y)):
     16         grad_y[i] = 1.
---> 17         grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
     18         jac.append(grad_x.reshape(x.shape))
     19         grad_y[i] = 0.

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
    155     return Variable._execution_engine.run_backward(
    156         outputs, grad_outputs, retain_graph, create_graph,
--> 157         inputs, allow_unused)
    158
    159

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2]] is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Output:
I sincerely thought this was a bug in PyTorch, but after posting a bug, I got a good answer from albanD. https://github.com/pytorch/pytorch/issues/36903#issuecomment-616671247 He also pointed out that https://discuss.pytorch.org/ is available for asking questions.

The problem arises because we traverse the computation graph several times again and again. Exactly what is going on here is beyond me though...

The in place edit that your error message refers to are the obvious ones: grad_y[i] = 1. and grad_y[i] = 0.. Resuing grad_y over and over again in the computation is what causes trouble. Redefining jacobian(...) as below works for me.

def jacobian(y, x, create_graph=False):
    jac = []
    flat_y = y.reshape(-1)
    for i in range(len(flat_y)):
        grad_y = torch.zeros_like(flat_y)
        grad_y[i] = 1.
        grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
        jac.append(grad_x.reshape(x.shape))
    return torch.stack(jac).reshape(y.shape + x.shape)

An alternative, that works, but is more like black magic to me is leaving jacobian(...) as it is, and instead redefine f(x) to

def f(x):
    return x * x * 1

That works too.


Instruction: Artificial Neural Networks | Which loss function is used for the optimization of the model?

Input:
I have a code sample from github. Link is here: github code sample

My question is "Which loss function is used for the optimization of the model?"

Adam Optimizer or CrossEntropyLoss

What are the differences between them?

Output:
Loss Function determines, how good is the model. It is done by comparing the predictions and actual. CrossEntropyLoss is commonly used for classification task.

Once Loss is identified. To train neural network model. we need to update weights of model.

It is done by calculating the gradients using the calculated loss. Where each optimizer determines how the loss has to be moved inside model. where Adam is one of the popular optimizer. It is based on momentum and RMSprop.


Instruction: Problem with updating running_mean and running_var in a custom Batchnorm built in Pytorch?

Input:
I have been trying to implement a custom batch normalization function such that it can be extended to the Multi GPU version, in particular, the DataParallel module in Pytorch.The custom batchnorm works alright when using 1 GPU, but, when extended to 2 or more, the running mean and variance work in the forward function, but when it returns back from the network, the mean and variance are reinitialized to 0 and 1.

The torch.nn.DataParallel mentions in the warning section that " In each forward, module is replicated on each device, so any updates to the running module in forward will be lost. For example, if module has a counter attribute that is incremented in each forward, it will always stay at the initial value because the update is done on the replicas which are destroyed after forward." But I am not really sure how to retain the mean and variance from the default device.

I have provided code with the result obtained during multi GPU training. This code utilizes the Batchnorm provided here.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.backends.cudnn as cudnn
import torchvision
import torchvision.transforms as transforms
from torch.nn.parameter import Parameter

class ptrblck_BatchNorm2d(nn.BatchNorm2d):
    def __init__(self, num_features, eps=1e-5, momentum=0.1,
                 affine=True, track_running_stats=True):
        super(ptrblck_BatchNorm2d, self).__init__(
            num_features, eps, momentum, affine, track_running_stats)

    def forward(self, input):
        self._check_input_dim(input)

        exponential_average_factor = 0.0

        if self.training and self.track_running_stats:
            if self.num_batches_tracked is not None:
                self.num_batches_tracked += 1
                if self.momentum is None:  # use cumulative moving average
                    exponential_average_factor = 1.0 / float(self.num_batches_tracked)
                else:  # use exponential moving average
                    exponential_average_factor = self.momentum

        # calculate running estimates
        if self.training:
            mean = input.mean([0, 2, 3])
            # use biased var in train
            var = input.var([0, 2, 3], unbiased=False)
            n = input.numel() / input.size(1)
            with torch.no_grad():
                self.running_mean = exponential_average_factor * mean\
                    + (1 - exponential_average_factor) * self.running_mean
                # update running_var with unbiased var
                self.running_var = exponential_average_factor * var * n / (n - 1)\
                    + (1 - exponential_average_factor) * self.running_var
        else:
            mean = self.running_mean
            var = self.running_var

        input = (input - mean[None, :, None, None]) / (torch.sqrt(var[None, :, None, None] + self.eps))
        if self.affine:
            input = input * self.weight[None, :, None, None] + self.bias[None, :, None, None]

        return input

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)
        self.bn1 = ptrblck_BatchNorm2d(64)
        print("==> printing bn1 mean when init")
        print(self.bn1.running_mean)
        print("==> printing bn1 when init")
        print(self.bn1.running_mean)
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.classifier = nn.Linear(64, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.relu(x)
        x = self.pool(x)
        x = self.avgpool(x)

        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        print("======================================================")
        print("==> printing bn1 running mean from NET during forward")
        print(net.module.bn1.running_mean)
        print("==> printing bn1 running mean from SELF. during forward")
        print(self.bn1.running_mean)
        print("==> printing bn1 running var from NET during forward")
        print(net.module.bn1.running_var)
        print("==> printing bn1 running mean from SELF. during forward")
        print(self.bn1.running_var)
        return x

# Data
print('==> Preparing data..')
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform_test)
testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=False, num_workers=2)
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

# Model
print('==> Building model..')
net = net()
net = torch.nn.DataParallel(net).cuda()
print('Number of GPU {}'.format(torch.cuda.device_count()))

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)

# Training
def train(epoch):
    print('\nEpoch: %d' % epoch)
    net.train()
    train_loss = 0
    correct = 0
    total = 0

    for batch_idx, (inputs, targets) in enumerate(trainloader):
        inputs, targets = inputs.cuda(), targets.cuda()
        outputs = net(inputs)
        loss = criterion(outputs, targets)
        print("====================================================")
        print("==> printing bn1 running mean FROM net after forward")
        print(net.module.bn1.running_mean)
        print("==> printing bn1 running var FROM net after forward")
        print(net.module.bn1.running_var)

        break
        # optimizer.zero_grad()
        # loss.backward()
        # optimizer.step()

        # train_loss += loss.item()
        # _, predicted = outputs.max(1)
        # total += targets.size(0)
        # correct += predicted.eq(targets).sum().item()

        # break

for epoch in range(0, 1):
    train(epoch)

Result:

==> Preparing data..
Files already downloaded and verified
Files already downloaded and verified
==> Building model..
==> printing bn1 mean when init
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
==> printing bn1 when init
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
Number of GPU 2

Epoch: 0
======================================================
==> printing bn1 running mean from NET during forward
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0')
==> printing bn1 running mean from SELF. during forward
tensor([ 0.0053,  0.0010, -0.0077, -0.0290,  0.0241,  0.0258, -0.0048,  0.0151,
        -0.0133,  0.0080,  0.0197, -0.0042, -0.0188,  0.0233,  0.0310, -0.0230,
        -0.0133,  0.0222,  0.0119, -0.0042, -0.0220, -0.0169, -0.0342, -0.0025,
         0.0338, -0.0070,  0.0202,  0.0050,  0.0108,  0.0008,  0.0363,  0.0347,
        -0.0106,  0.0082,  0.0128,  0.0074,  0.0111, -0.0030, -0.0089,  0.0070,
        -0.0262, -0.0029,  0.0053, -0.0136, -0.0183,  0.0045, -0.0014, -0.0221,
         0.0132,  0.0064,  0.0388, -0.0220, -0.0008,  0.0400, -0.0187,  0.0397,
        -0.0131, -0.0176,  0.0035,  0.0055, -0.0270,  0.0066, -0.0149,  0.0135],
       device='cuda:0')
==> printing bn1 running var from NET during forward
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], device='cuda:0')
==> printing bn1 running mean from SELF. during forward
tensor([0.9665, 0.9073, 0.9220, 1.0947, 1.0687, 0.9624, 0.9252, 0.9131, 0.9066,
        0.9536, 0.9258, 0.9203, 1.0359, 0.9690, 1.1066, 1.0636, 0.9135, 0.9644,
        0.9373, 0.9846, 0.9696, 0.9454, 1.0459, 0.9245, 0.9778, 0.9709, 0.9352,
        0.9995, 0.9657, 0.9510, 1.0943, 1.0171, 0.9298, 1.0747, 0.9341, 0.9635,
        0.9978, 0.9303, 0.9261, 0.9137, 0.9569, 1.0066, 1.0463, 0.9955, 0.9621,
        0.9172, 0.9836, 0.9817, 0.9086, 0.9576, 1.0905, 0.9861, 0.9661, 1.1773,
        0.9345, 1.0904, 0.9133, 1.0660, 0.9164, 0.9058, 0.9446, 0.9225, 1.0914,
        0.9292], device='cuda:0')
======================================================
==> printing bn1 running mean from NET during forward
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0')
==> printing bn1 running mean from SELF. during forward
tensor([-0.0020,  0.0002, -0.0103, -0.0426,  0.0386,  0.0311, -0.0059,  0.0151,
        -0.0140,  0.0145,  0.0218, -0.0029, -0.0281,  0.0284,  0.0449, -0.0329,
        -0.0107,  0.0278,  0.0135, -0.0123, -0.0260, -0.0214, -0.0423, -0.0035,
         0.0410, -0.0097,  0.0276,  0.0102,  0.0197, -0.0001,  0.0483,  0.0451,
        -0.0078,  0.0190,  0.0135, -0.0004,  0.0196, -0.0028, -0.0140,  0.0070,
        -0.0332, -0.0110,  0.0151, -0.0210, -0.0226,  0.0074, -0.0088, -0.0314,
         0.0125, -0.0003,  0.0505, -0.0312,  0.0086,  0.0544, -0.0245,  0.0528,
        -0.0086, -0.0290,  0.0063,  0.0042, -0.0339,  0.0061, -0.0277,  0.0092],
       device='cuda:1')
==> printing bn1 running var from NET during forward
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], device='cuda:0')
==> printing bn1 running mean from SELF. during forward
tensor([0.9665, 0.9072, 0.9211, 1.0999, 1.0714, 0.9610, 0.9209, 0.9125, 0.9063,
        0.9553, 0.9260, 0.9189, 1.0386, 0.9706, 1.1139, 1.0610, 0.9121, 0.9660,
        0.9366, 0.9886, 0.9683, 0.9454, 1.0511, 0.9227, 0.9792, 0.9704, 0.9330,
        0.9989, 0.9657, 0.9476, 1.1008, 1.0191, 0.9294, 1.0814, 0.9320, 0.9642,
        1.0006, 0.9287, 0.9254, 0.9128, 0.9559, 1.0100, 1.0521, 0.9972, 0.9621,
        0.9168, 0.9849, 0.9803, 0.9083, 0.9556, 1.0946, 0.9865, 0.9651, 1.1880,
        0.9330, 1.0959, 0.9116, 1.0706, 0.9149, 0.9057, 0.9450, 0.9215, 1.0972,
        0.9261], device='cuda:1')
====================================================
==> printing bn1 running mean FROM net after forward
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0')
==> printing bn1 running var FROM net after forward
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], device='cuda:0')

How can I make sure that the running estimates of the default device be used? Currently, I am not working towards synchronized Batchnorm.

Output:
Replacing

self.running_mean = (...)

with

self.running_mean.copy_(...)

did the job.

Reference


Instruction: Index a torch tensor with an array

Input:
I have the following torch tensor:

tensor([[-0.2,  0.3],
    [-0.5,  0.1],
    [-0.4,  0.2]])

and the following numpy array: (I can convert it to something else if necessary)

[1 0 1]

I want to get the following tensor:

tensor([0.3, -0.5, 0.2])

i.e. I want the numpy array to index each sub-element of my tensor. Preferably without using a loop.

Thanks in advance

Output:
You may want to use torch.gather - "Gathers values along an axis specified by dim."

t = torch.tensor([[-0.2,  0.3],
    [-0.5,  0.1],
    [-0.4,  0.2]])
idxs = np.array([1,0,1])

idxs = torch.from_numpy(idxs).long().unsqueeze(1)
# or   torch.from_numpy(idxs).long().view(-1,1)

t.gather(1, idxs)
tensor([[ 0.3000],
        [-0.5000],
        [ 0.2000]])

Here, your index is numpy array so you have to convert it to LongTensor.


Instruction: PyTorch - what is the reason to resize my image and how do I decide the best size?

Input:
I have some malaria infected blood cell images (multiple blood cells per image) with details of:

num_features = 929
num_features_train = 743
num_features_test = 186
depth = 24
channels = 3
width = 1600
height = 1200

In the previous image classification I did on a malaria infected blood cell image datasets (single blood cell per image) the image details were width = 100, height = 101, depth = 24.  So resizing to 50x50 didn't seem an issue.

I need to resize this, obviously, but don't know how to choose the best size for resizing an image so large.  I can't find anything in my online searching talking about this.  Any advise/experience would be helpful and greatly appreciated.  Thanks!!

p.s. I already figured out that if I don't resize something this large I will get a memory error. Yeah, probably. :)
MemoryError: Unable to allocate 6.64 GiB for an array with shape (929, 1600, 1600, 3) and data type uint8

p.p.s. resized to 100x100 and still got memory error
       resized to 50x50 and it was OK

So, I guess my question is, doesn't reducing the size so much reduce the resolution?  So how does the convolutional layer filters do proper filtering if the resolution is reduced so drastically?

Output:
Reducing the size reduces the resolution, but it still can keep all important features of the original image. Smaller images = fewer features = quicker training, less overfishing. However, a too drastic drop in size may cause images to lose the point of interest. For example, after resizing, a tumor may be smoothened by the surrounding pixels and disappear.

Overall: if images keep the point of interest after resizing, it should be OK.


Instruction: libtorch (PyTorch C++) weird class syntax

Input:
In the official PyTorch C++ examples on GitHub Here
you can witness a strange definition of a class:

class CustomDataset : public torch::data::datasets::Dataset<CustomDataset> {...}

My understanding is that this defines a class CustomDataset which "inherits from" or "extends" torch::data::datasets::Dataset<CustomDataset>. This is weird to me since the class we're creating is inheriting from another class which is parameterized by the class we're creating...How does this even work? What does it mean? This seems to me like an Integer class inheriting from vector<Integer>, which seems absurd.

Output:
This is the curiously-recurring template pattern, or CRTP for short. A major advantage of this technique is that it enabled so-called static polymorphism, meaning that functions in torch::data::datasets::Dataset can call into functions of CustomDataset, without needing to make those functions virtual (and thus deal with the runtime mess of virtual method dispatch and so on). You can also perform compile-time metaprogramming such as compile-time enable_ifs depending on the properties of the custom dataset type.

In the case of PyTorch, BaseDataset (the superclass of Dataset) uses this technique heavily to support operations such as mapping and filtering:

  template <typename TransformType>
  MapDataset<Self, TransformType> map(TransformType transform) & {
    return datasets::map(static_cast<Self&>(*this), std::move(transform));
  }

Note the static cast of this to the derived type (legal as long as CRTP is properly applied); datasets::map constructs a MapDataset object which is also parametrized by the dataset type, allowing the MapDataset implementation to statically call methods such as get_batch (or encounter a compile-time error if they do not exist).

Furthermore, since MapDataset receives the custom dataset type as a type parameter, compile-time metaprogramming is possible:

  /// The implementation of `get_batch()` for the stateless case, which simply
  /// applies the transform to the output of `get_batch()` from the dataset.
  template <
      typename D = SourceDataset,
      typename = torch::disable_if_t<D::is_stateful>>
  OutputBatchType get_batch_impl(BatchRequestType indices) {
    return transform_.apply_batch(dataset_.get_batch(std::move(indices)));
  }

  /// The implementation of `get_batch()` for the stateful case. Here, we follow
  /// the semantics of `Optional.map()` in many functional languages, which
  /// applies a transformation to the optional's content when the optional
  /// contains a value, and returns a new optional (of a different type)  if the
  /// original optional returned by `get_batch()` was empty.
  template <typename D = SourceDataset>
  torch::enable_if_t<D::is_stateful, OutputBatchType> get_batch_impl(
      BatchRequestType indices) {
    if (auto batch = dataset_.get_batch(std::move(indices))) {
      return transform_.apply_batch(std::move(*batch));
    }
    return nullopt;
  }

Notice that the conditional enable is dependent on SourceDataset, which we only have available because the dataset is parametrized with this CRTP pattern.


Instruction: Resize Vs CenterCrop Vs RandomResizedCrop Vs RandomCrop

Input:
Can anyone tell me in which situations the above functions are used and how they affect the image size?
I want to resize the Cat V Dogs images and i am a bit confuse about how to use them.

Output:
There are lots of details in TorchVision documentation actually.
The typical use case is for object detection or image segmentation tasks, but other uses could exist.
Here is a non-exhaustive list of uses:

Resize is used in Convolutional Neural Networks to adapt the input image to the network input shape, in this case this is not data-augmentation but just pre-processing. It can also be used in Fully Convolutional Networks to emulate different scales for an input image, this is data-augmentation.
CenterCrop RandomCrop and RandomResizedCrop are used in segmentation tasks to train a network on fine details without impeding too much burden during training. For with a database of 2048x2048 images you can train on 512x512 sub-images and then at test time infer on full resolution images. It is also used in object detection networks as data-augmentation. The resized variant lets you combine the previous resize operation.

All of them potentially change the image resolution.


Instruction: Gradient of the loss of DistilBERT for measuring token importance

Input:
I am trying to access the gradient of the loss in DistilBERT with respect to each attention weight in the first layer. I could access the computed gradient value of the output weight matrix via the following code when requires_grad=True

loss.backward()
for name, param in model.named_parameters():
    if name == 'transformer.layer.0.attention.out_lin.weight':
       print(param.grad)  #shape is [768,768]

where model is the loaded distilbert model.
My question is how to get the gradient with respect to [SEP] or [CLS] or other tokens' attention? I need it to reproduce the figure about the "Gradient-based feature importance estimates for attention to [SEP]" in the following link:
https://medium.com/analytics-vidhya/explainability-of-bert-through-attention-7dbbab8a7062

A similar question for the same purpose has been asked in the following, but it is not my issue:
BERT token importance measuring issue. Grad is none

Output:
By default, the gradients are retained only for parameters, basically just to save memory. If you need gradients of inner nodes of the computation graph, you need to have the respective tensor before calling backward() and add a hook that will be executed at the backward pass.

A minimum solution from PyTorch forum:

yGrad = torch.zeros(1,1)
def extract(xVar):
    global yGrad
    yGrad = xVar

xx = Variable(torch.randn(1,1), requires_grad=True)
yy = 3*xx
zz = yy**2

yy.register_hook(extract)

#### Run the backprop:
print (yGrad) # Shows 0.
zz.backward()
print (yGrad) # Show the correct dzdy

In this case, the gradients are stored in a global variable where they persist after PyTorch get rid of them in the graph itself.


Instruction: how to solve size mismatch for this model?

Input:
data_transforms = {
'train': transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
'val': transforms.Compose([
    transforms.Resize(224),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),

batchsize=4

class Net(nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 64, kernel_size=5, padding=1)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(64, 128, kernel_size=5, padding=1)
    self.fc1 = nn.Linear(256*6*6, 4096)
    self.fc2 = nn.Linear(4096, 4096)
    self.fc3 = nn.Linear(4096, 2)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.reshape(x.size(0), -1)
    print(x.shape)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

RuntimeError: size mismatch, m1: [4 x 373248], m2: [9216 x 4096] at C:/w/1/s/tmp_conda_3.8_075429/conda/conda-bld/pytorch_1579852542185/work/aten/src\THC/generic/THCTensorMathBlas.cu:290

Output:
With the provided debug information we can only tell that there is a size mismatch in one of the layers.

Looking at the error, it seems the error is in the first linear layer. You should change its size to 373248 instead of 256*6*6.

This should also be clear from the output of the print statement print(x.shape).


Instruction: softmax dims and variable volatile in PyTorch

Input:
I have a code for previous version of PyTorch and I receive 2 warning for the 3nd line of it:

import torch.nn.functional as F

def select_action(self, state):
        probabilities = F.softmax(self.model(Variable(state, volatile = True))*100) # T=100
        action = probs.multinomial(num_samples=1)
        return action.data[0,0]

  UserWarning: volatile was removed and now has no effect. Use with
  torch.no_grad(): instead.

  UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X > as an argument.

I found that:

  Volatile is recommended for purely inference mode, when you’re sure
  you won’t be even calling .backward(). It’s more efficient than any
  other autograd setting - it will use the absolute minimal amount of
  memory to evaluate the model. volatile also determines that
  requires_grad is False.

Am I right that I should just remove it?
And because I want to get probabilities therefore should I use dim=1 ?
and the 3nd line of my code should look like:

    probabilities = F.softmax(self.model(Variable(state), dim=1)*100) # T=100

state is created here:

def update(self, reward, new_signal):
   new_state = torch.Tensor(new_signal).float().unsqueeze(0)
   self.memory.push((self.last_state, new_state, torch.LongTensor([int(self.last_action)]), torch.Tensor([self.last_reward])))
   action = self.select_action(new_state)
   if len(self.memory.memory) > 100:
       batch_state, batch_next_state, batch_action, batch_reward = self.memory.sample(100)
       self.learn(batch_state, batch_next_state, batch_reward, batch_action)
   self.last_action = action
   self.last_state = new_state
   self.last_reward = reward
   self.reward_window.append(reward)
   if len(self.reward_window) > 1000:
       del self.reward_window[0]
   return action

Output:
You are right but not "fully" right.

Except changes you mentioned you should use torch.no_grad() as mentioned like this:

def select_action(self, state):
    with torch.no_grad():
        probabilities = F.softmax(self.model(state), dim=1)*100
        action = probs.multinomial(num_samples=1)
        return action.data[0,0]

This block turns off autograd engine for code within it (so you save the memory similarly to volatile).

Also please notice Variable is deprecated as well (check here) and state should be simply torch.tensor created with requires_grad=True.

BTW. You have probs and probabilities but I assume it's the same thing and merely a typo.


Instruction: Is it safe to always use torch.tensor or torch.FloatTensor? Or do I need to treat Ints with care?

Input:
  https://pytorch.org/docs/stable/tensors.html

I am trying to understand the difference between tensor, FloatTensor, IntTensor - and am wondering if I can just always stick to tensor... or maybe FloatTensor.

I am going to be using a mix of different tensors that will be:

{integers:labels, floats:continuous, one-hot-encoded:categoricals}

Do I need to explicitly set each of these as kinds of variables as different types of tensors? Will they all work as floats? Will they work in combination with each other?

Would this get me into trouble downstream?

l_o_l = [[1,2,3],[1,2,3],[1,2,3]]

int_tnz = th.FloatTensor(l_o_l)
int_tnz

tensor([[1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.]])

int_tnz.dtype

torch.float32

l_o_fl = [[1.1,2.2,3.3],[1.1,2.2,3.3],[1.1,2.2,3.3]]

int_tnz = th.tensor(l_o_fl)
int_tnz

tensor([[1.1000, 2.2000, 3.3000],
        [1.1000, 2.2000, 3.3000],
        [1.1000, 2.2000, 3.3000]])

int_tnz.dtype

torch.float32

Output:
CrossEntropyLoss (or NLLLoss) expect the target type to be Long. For example, the code below raises a RuntimeError:

import torch.nn

criterion = torch.nn.CrossEntropyLoss()

predicted = torch.rand(10, 10, dtype=torch.float)
target = torch.rand(10) #.to(torch.long)
criterion(predicted, target)
# RuntimeError: expected scalar type Long but found Float

You need to uncomment the conversion to make it work. I cannot think of a bigger issue, but why bother converting the integers to float in the first place?

Regarding the use of torch.tensor and torch.FloatTensor, I prefer the former. torch.FloatTensor seems to be the legacy constructor, and it does not accept device as an argument. Again, I do not think this a big concern, but still, using torch.tensor increases the readability of the code.


Instruction: bool value of Tensor with more than one value is ambiguous

Input:
I'm writing a neural network to do regression and here is my codes:

class Model(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.h1 = nn.Linear(input_size, hidden_size)
        self.h2 = nn.Linear(hidden_size, hidden_size)
        self.h3 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        x = self.h1(x)
        x = Fuc.tanh(x)
        x = self.h2(x)
        x = Fuc.relu(x)
        x = self.h3(x)
        return x

model = Model(input_size=input_size, hidden_size=hidden_size, num_classes=num_classes)
opt = optim.Adam(params=model.parameters(), lr=learning_rate)

for epoch in range(1000):
    out = model(data)
    print('target', target)
    print('pred', out)
    loss = torch.nn.MSELoss(out, target)
    print('loss', loss)

    model.zero_grad()
    loss.backward()
    opt.step()

my input is of shape (numberOfSample X 2) and out put is of the form [[2],[3],...], namely a list of lists where each inner list contain one number.

Ok so Now I train the neural network and got this error:

       ...
       [-0.1753],
        [-0.1753],
        [-0.1753]], grad_fn=<AddmmBackward>)
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1340: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
  warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-26-38e8026bfe54> in <module>()
     68     print('target', target)
     69     print('pred', out)
---> 70     loss = torch.nn.MSELoss(out, target)
     71     print('loss', loss)
     72

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/_reduction.py in legacy_get_string(size_average, reduce, emit_warning)
     34         reduce = True
     35
---> 36     if size_average and reduce:
     37         ret = 'mean'
     38     elif reduce:

RuntimeError: bool value of Tensor with more than one value is ambiguous

Output:
The issue originates from calling torch.nn.MSELoss(out, target) which is the constructor for the MSELoss which accepts size_average and reduce as the first and second optional positional arguments.

loss = torch.nn.MSELoss(out, target)

Instead, you need to create an MSELoss object first and pass the out and the target to that object.

criterion = torch.nn.MSELoss()

for epoch in range(1000):
    out = model(data)
    loss = criterion(out, target)
    loss.backward()


Instruction: how to set values for layers in pytorch nn.module?

Input:
I have a model that I am trying to get working.  I am working through the errors, but now I think it has come down to the values in my layers.  I get this error:

RuntimeError: Given groups=1, weight of size 24 1 3 3, expected input[512, 50, 50, 3] to have 1 channels,
but got 50 channels instead

My parameters are:

LR = 5e-2
N_EPOCHS = 30
BATCH_SIZE = 512
DROPOUT = 0.5

My image information is:

depth=24
channels=3
original height = 1600
original width = 1200
resized to 50x50

This is the size of my data:

Train shape (743, 50, 50, 3) (743, 7)
Test shape (186, 50, 50, 3) (186, 7)
Train pixels 0 255 188.12228712427097 61.49539262385051
Test pixels 0 255 189.35559211469533 60.688278787628775

I looked here to try to understand what values each layer is expecting, but when I put in what it says here, https://towardsdatascience.com/pytorch-layer-dimensions-what-sizes-should-they-be-and-why-4265a41e01fd, it gives me errors about wrong channels and kernels.

I found torch_summary to give me more understanding about the outputs, but it only poses more questions.

This is my torch_summary code:

from torchvision import models
from torchsummary import summary
import torch
import torch.nn as nn

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(1,24, kernel_size=5)  # output (n_examples, 16, 26, 26)
        self.convnorm1 = nn.BatchNorm2d(24) # channels from prev layer
        self.pool1 = nn.MaxPool2d((2, 2))  # output (n_examples, 16, 13, 13)
        self.conv2 = nn.Conv2d(24,48,kernel_size=5)  # output (n_examples, 32, 11, 11)
        self.convnorm2 = nn.BatchNorm2d(48) # 2*channels?
        self.pool2 = nn.AvgPool2d((2, 2))  # output (n_examples, 32, 5, 5)
        self.linear1 = nn.Linear(400,120)  # input will be flattened to (n_examples, 32 * 5 * 5)
        self.linear1_bn = nn.BatchNorm1d(400) # features?
        self.drop = nn.Dropout(DROPOUT)
        self.linear2 = nn.Linear(400, 10)
        self.act = torch.relu

    def forward(self, x):
        x = self.pool1(self.convnorm1(self.act(self.conv1(x))))
        x = self.pool2(self.convnorm2(self.act(self.conv2(x))))
        x = self.drop(self.linear1_bn(self.act(self.linear1(x.view(len(x), -1)))))
        return self.linear2(x)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model=CNN().to(device)
summary(model, (3, 50, 50))

This is what it gave me:

  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 24 1 5 5, expected input[2, 3, 50, 50] to have 1 channels, but got 3 channels instead

When I run my whole code, and unsqueeze_(0) my data, like so....x_train = torch.from_numpy(x_train).unsqueeze_(0)  I get this error:

 File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 24 1 5 5, but got 5-dimensional input of size [1, 743, 50, 50, 3] instead

I don't know how to figure out how to fill in the proper values in the layers.  Will someone please help me find the correct values and understand how to understand this?  I do know that the output of one layer should be the input of another layer.  Nothing is matching up with what I thought I knew.
Thanks in advance!!

Output:
It seems you have wrong order of input x tensor axis.
As you can see in the doc Conv2d input must be (N, C, H, W)

  N is a batch size, C denotes a number of channels, H is a height of input planes in pixels, and W is width in pixels.

So, To make it right use torch.permute to swap axis in forward pass.

...
def forward(self, x):
    x = x.permute(0, 3, 1, 2)
    ...
    ...
    return self.linear2(x)
...

Example of permute:

t = torch.rand(512, 50, 50, 3)
t.size()
torch.Size([512, 50, 50, 3])

t = t.permute(0, 3, 1, 2)
t.size()
torch.Size([512, 3, 50, 50])


Instruction: Pytorch: Increase indice of tensor to specific size

Input:
I have a tensor of [20, 3, 32, 32]
I want to increase the 3 to 64 ([20,64,32,32], where 20 is the batch size). I tried the repeat function. But that only gives me 63 or 66, because with repeat you can only tile (multiply) indices.
How can I solve this?
Thanks!

Output:
You can repeat using torch.repeat and then filter with indices.

t = torch.rand(20,3,32,32)
t.shape
torch.Size([20, 3, 32, 32])

t = t.repeat(1,22,1,1)[:,:-2,:,:]
t.shape
torch.Size([20, 64, 32, 32])


Instruction: How to know which gcc version to choose when compiling pytorch from source?

Input:
I am working on installing PyTorch from source but am unsure about the specific dependency versions to use for the version of PyTorch I want to install.

In particular, I noticed performance variations depending on the gcc version I use to compile PyTorch. Which compiler should I be using to get the best PyTorch performance?

Tensorflow doc provides such useful information. They call it "Tested build configurations":
https://www.tensorflow.org/install/source#tested_build_configurations.

Output:
The README.md has instructions to build from source.

If you are installing from source, you will need a C++14 compiler. Also, we highly recommend installing an Anaconda environment. You will get a high-quality BLAS library (MKL) and you get controlled dependency versions regardless of your Linux distro.

Once you have Anaconda installed, here are the instructions.

If you want to compile with CUDA support, install

NVIDIA CUDA 9 or above
NVIDIA cuDNN v7 or above
Compiler compatible with CUDA

If you want to disable CUDA support, export environment variable USE_CUDA=0. Other potentially useful environment variables may be found in setup.py.

If you are building for NVIDIA's Jetson platforms (Jetson Nano, TX1, TX2, AGX Xavier), Instructions to install PyTorch for Jetson Nano are available here

You can found Latest, official Compiler requirements here


Instruction: Train Accuracy increases, Train loss is stable, Validation loss Increases, Validation Accuracy is low and increases

Input:
My neural network trainign in pytorch is getting very wierd.

I am training a known dataset that came splitted into train and validation.
I'm shuffeling the data during training and do data augmentation on the fly.

I have those results:

Train accuracy start at 80% and increases

Train loss decreases and stays stable

Validation accuracy start at 30% but increases slowly

Validation loss increases

I have the following graphs to show:

How can you explain that the validation loss increases and the validation accuracy increases?
How can be such a big difference of accuracy between validation and training sets?  90% and 40%?

Update:

I balanced the data set.
It is binary classification. It now has now 1700 examples from class 1, 1200 examples from class 2. Total 600 for validation and 2300 for training.
I still see similar behavior:

**Can it be becuase I froze the weights in part of the network?

**Can it be becuase the hyperparametrs like lr?

Output:
I found the solution:
I had different data augmentation for training set and validation set. Matching them also increased the validation accuracy!


Instruction: Changing thresholds in the Sigmoid Activation in Neural Networks

Input:
Hi I am new to machine learning and I have a query about changing thresholds for sigmoid function.

I know Sigmoid function's value is in the range [0;1], 0.5 is taken as a threshold, if h(theta) < 0.5 we assume that it's value is 0, if h(theta) >= 0.5 then it's 1.

Thresholds are used only on the output layer of the network and it's only when classifying. So, if you're trying to classify between 3 classes can you give different thresholds for each class (0.2,0.4,0.4 - for each class)? Or can you specify a different threshold overall, like 0.8? I am unsure how to define this in the code below. Any guidance is appreciated.

# Hyper Parameters
input_size = 14
hidden_size = 40
hidden_size2 = 30
num_classes = 3
num_epochs = 600
batch_size = 34
learning_rate = 0.01

class Net(torch.nn.Module):
    def __init__(self, n_input, n_hidden, n_hidden2, n_output):
        super(Net, self).__init__()
        # define linear hidden layer output
        self.hidden = torch.nn.Linear(n_input, n_hidden)
        self.hidden2 = torch.nn.Linear(n_hidden, n_hidden2)
        # define linear output layer output
        self.out = torch.nn.Linear(n_hidden, n_output)

    def forward(self, x):
        """
            In the forward function we define the process of performing
            forward pass, that is to accept a Variable of input
            data, x, and return a Variable of output data, y_pred.
        """
        # get hidden layer input
        h_input1 = self.hidden(x)
        # define activation function for hidden layer
        h_output1 = torch.sigmoid(h_input1)

        # get hidden layer input
        h_input2 = self.hidden2(h_output1)
        # define activation function for hidden layer
        h_output2 = torch.sigmoid(h_input2)

        # get output layer output
        out = self.out(h_output2)

        return out

net = Net(input_size, hidden_size, hidden_size, num_classes)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

all_losses = []

for epoch in range(num_epochs):
    total = 0
    correct = 0
    total_loss = 0
    for step, (batch_x, batch_y) in enumerate(train_loader):
        X = batch_x
        Y = batch_y.long()

    # Forward + Backward + Optimize
    optimizer.zero_grad()  # zero the gradient buffer
    outputs = net(X)
    loss = criterion(outputs, Y)
    all_losses.append(loss.item())
    loss.backward()
    optimizer.step()

    if epoch % 50 == 0:
        _, predicted = torch.max(outputs, 1)
        # calculate and print accuracy
        total = total + predicted.size(0)
        correct = correct + sum(predicted.data.numpy() == Y.data.numpy())
        total_loss = total_loss + loss
    if epoch % 50 == 0:
        print(
            "Epoch [%d/%d], Loss: %.4f, Accuracy: %.2f %%"
            % (epoch + 1, num_epochs, total_loss, 100 * correct / total)
        )

train_input = train_data.iloc[:, :input_size]
train_target = train_data.iloc[:, input_size]

inputs = torch.Tensor(train_input.values).float()
targets = torch.Tensor(train_target.values - 1).long()

outputs = net(inputs)
_, predicted = torch.max(outputs, 1)

Output:
You can use any threshold you find suitable.

Neural networks are known to be often over-confident (e.g. applying 0.95 to one of 50 classes), so it may be beneficial to use different threshold in your case.

Your training is fine, but you should change predictions (last two lines) and use torch.nn.softmax like this:

outputs = net(inputs)
probabilities = torch.nn.functional.softmax(outputs, 1)

As mentioned in other answer you will get each row with probabilities summing to 1 (previously you had unnormalized probabilities a.k.a. logits).

Now, just use your desired threshold on those probabilities:

predictions = probabilities > 0.8

Please notice you may get only zeros in some cases (e.g. [0.2, 0.3, 0.5]).

This would mean neural network isn't confident enough according to your standards and would probably drop number of incorrect positive predictions (abstract, but say you are predicting whether a patient doesn't have one of mutually exclusive 3 diseases. It's better to say so only if you are really sure).

Different thresholds for each class

This could be done as well like this:

thresholds = torch.tensor([0.1, 0.1, 0.8]).unsqueeze(0)
predictions = probabilities > thresholds

Final comments

Please notice in case of softmax only one class should be the answer (as pointed out in another answer) and this approach (and mention of sigmoid) may indicate you are after multilabel classification.

If you want to train your network so it can simultaneously predict classes you should use sigmoid and change your loss to torch.nn.BCEWithLogitsLoss.


Instruction: Why is PIL used so often with Pytorch?

Input:
I noticed that a lot of dataloaders use PIL to load and transform images, e.g. the dataset builders in torchvision.datasets.folder.

My question is: why use PIL? You would need to do an np.asarray operation before turning it into a tensor. OpenCV seems to load it directly as a numpy array, and is faster too.

One reason I can think of is because PIL has a rich transforms library, but I feel like several of those transforms can be quickly implemented.

Output:
There is a discussion about adding OpenCV as one of possible backends in torchvision PR.

In summary, some reasons provided:

OpenCV2 loads images in BGR format which would require wrapper class to handle changing to RGB internally or format of loaded images backend dependent
This in turn would lead to code duplication in functional transforms in torchvision many of which use PIL operations (as transformations to support multiple backends would be pretty convoluted)
OpenCV loads images as np.array, it's not really easier to do transformations on arrays
Different representation might lead to hard to catch by users bugs
PyTorch's modelzoo is dependent on RGB format as well and they would like to have it easily supported
Doesn't play well with Python's multiprocessing (but it's no-issue as it was an issue for Python 2)

To be honest I don't see much movement towards this idea as there exists albumentations which uses OpenCV and can be integrated with PyTorch rather smoothly.

A little off-topic, but one can choose faster backend via torchvision.set_image_backend to Intel's accimage. Also Pillow-SIMD can be used as a drop-in replacement for PIL (it is supposedly faster and recommended by fastai project).

When it comes to performance benchmarks they do not seem too reliable and it's not that easy to tell AFAIK.


Instruction: What is the purpose of the dim parameter in torch.nn.Softmax

Input:
I don't understand to what does the dim parameter applies in torch.nn.Softmax. There is a warning that tells me to use it and I set it to 1, but I don't understand what I am setting. Where is it being used in the formula:

Softmax(xi)=exp(xi)/∑jexp(xj)

There is no dim here, so to what does it apply?

Output:
The Pytorch documentation on torch.nn.Softmax states:
dim (int) – A dimension along which Softmax will be computed (so every slice along dim will sum to 1).
For example, if you have a matrix with two dimensions, you can choose whether you want to apply the softmax to the rows or the columns:
import torch
import numpy as np

softmax0 = torch.nn.Softmax(dim=0) # Applies along columns
softmax1 = torch.nn.Softmax(dim=1) # Applies along rows

v = np.array([[1,2,3],
              [4,5,6]])
v =  torch.from_numpy(v).float()

softmax0(v)
# Returns
#[[0.0474, 0.0474, 0.0474],
# [0.9526, 0.9526, 0.9526]])

softmax1(v)
# Returns
#[[0.0900, 0.2447, 0.6652],
# [0.0900, 0.2447, 0.6652]]

Note how for softmax0 the columns add to 1, and for softmax1 the rows add to 1.


Instruction: While converting a PIL image into a tensor why the pixels are changing?

Input:
transform = transforms.Compose([transforms.ToPILImage(), transforms.ToTensor()])

Before applying the transformation

After applying the transformation

Q.1 Why the pixel values are changed?
Q.2 How to correct this?

Output:
I was able to solve this problem by normalizing the input data before transforming it.
The problem was that ToPILImage() was discarding all the values which were greater than 1 hence the bright pixels became dark.


Instruction: Getting error when using print() or summary() in pytorch to see the layers and weight dimensions in a Pytorch model

Input:
When using print on an existing model,
it doesn't print the model. Instead it shows:
<function resnext101_32x8d at 0x00000178CC26BA68>

>>> import torch
>>> import torchvision.models as models
>>> m1 = models.resnext101_32x8d
>>> print(m1)
<function resnext101_32x8d at 0x00000178CC26BA68>
>>>

When using summary, it gives the following error:

  AttributeError: 'function' object has no attribute 'apply'

>>> import torch
>>> import torchvision.models as models
>>> from torchvision import summary
>>> m1 = models.resnext101_32x8d
>>>
>>> summary(m1, (3, 224, 224))
 Traceback(most recent call last):
   File "<stdin>", line 1, in <module>
   File torchsummary.py, line 68, in summary
      model.apply(register_hook)
AttributeError: 'function' object has no attribute 'apply'

How to fix these issues related to print and summary? Any other ways to easily see all pytorch layers and model topology?

Output:
models.resnext101_32x8d is the class constructor, you need to call the constructor, just add parentheses at the end.

m1 = models.resnext101_32x8d()
print(m1)


Instruction: Pass pretrained weights in CNN Pytorch to a CNN in Tensorflow

Input:
I have trained this network in Pytorch for 224x224 size images and 4 classes.

class CustomConvNet(nn.Module):
    def __init__(self, num_classes):
        super(CustomConvNet, self).__init__()

        self.layer1 = self.conv_module(3, 64)
        self.layer2 = self.conv_module(64, 128)
        self.layer3 = self.conv_module(128, 256)
        self.layer4 = self.conv_module(256, 256)
        self.layer5 = self.conv_module(256, 512)
        self.gap = self.global_avg_pool(512, num_classes)
        #self.linear = nn.Linear(512, num_classes)
        #self.relu = nn.ReLU()
        #self.softmax = nn.Softmax()

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.layer5(out)
        out = self.gap(out)
        out = out.view(-1, 4)
        #out = self.linear(out)

        return out

    def conv_module(self, in_num, out_num):
        return nn.Sequential(
            nn.Conv2d(in_num, out_num, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2), stride=None))

    def global_avg_pool(self, in_num, out_num):
        return nn.Sequential(
            nn.Conv2d(in_num, out_num, kernel_size=3, stride=1, padding=1),
            #nn.BatchNorm2d(out_num),
            #nn.LeakyReLU(),

            nn.ReLU(),
            nn.Softmax(),
            nn.AdaptiveAvgPool2d((1, 1)))

I got the weights from the first Conv2D and it's size torch.Size([64, 3, 3, 3])

I have saved it as:

weightsCNN = net.layer1[0].weight.data
np.save('CNNweights.npy', weightsCNN)

This is my model I built in Tensorflow. I would like to pass those weights I saved from the Pytorch model into this Tensorflow CNN.

    model = models.Sequential()
    model.add(layers.Conv2D(64, (3, 3), activation='relu', input_shape=(224, 224, 3)))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(128, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(256, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(256, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(512, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(512, (3, 3), activation='relu'))

    model.add(layers.GlobalAveragePooling2D())
    model.add(layers.Dense(4, activation='softmax'))
    print(model.summary())

    adam = optimizers.Adam(learning_rate=0.0001, amsgrad=False)
    model.compile(loss='categorical_crossentropy',
                  optimizer=adam,
                  metrics=['accuracy'])

    nb_train_samples = 6596
    nb_validation_samples = 1290
    epochs = 10
    batch_size = 256

    history = model.fit_generator(
        train_generator,
        steps_per_epoch=np.ceil(nb_train_samples/batch_size),
        epochs=epochs,
        validation_data=validation_generator,
        validation_steps=np.ceil(nb_validation_samples / batch_size)
        )

How should I actually do that? What shape of weights does Tensorflow require? Thanks!

Output:
You can check shapes of all weights of all keras layers quite simply:

for layer in model.layers:
    print([tensor.shape for tensor in layer.get_weights()])

This would give you shapes of all weights (including biases), so you can prepare loaded numpy weights accordingly.

To set them, do something similar:

for torch_weight, layer in zip(model.layers, torch_weights):
    layer.set_weights(torch_weight)

where torch_weights should be a list containing lists of np.array which you would have to load.

Usually each element of torch_weights would contain one np.array for weights and one for bias.

Remember shapes received from print have to be exactly the same as the ones you put in set_weights.

See documentation for more info.

BTW. Exact shapes are dependent on layers and operations performed by model, you may have to transpose some arrays sometimes to "fit them in".


Instruction: optimizer got an empty parameter list (skorch)

Input:
So, I am used to use PyTorch and now decided to give Skorch a shot.

Here they define the network as

class ClassifierModule(nn.Module):
    def __init__(
            self,
            num_units=10,
            nonlin=F.relu,
            dropout=0.5,
    ):
        super(ClassifierModule, self).__init__()
        self.num_units = num_units
        self.nonlin = nonlin
        self.dropout = dropout

        self.dense0 = nn.Linear(20, num_units)
        self.nonlin = nonlin
        self.dropout = nn.Dropout(dropout)
        self.dense1 = nn.Linear(num_units, 10)
        self.output = nn.Linear(10, 2)

    def forward(self, X, **kwargs):
        X = self.nonlin(self.dense0(X))
        X = self.dropout(X)
        X = F.relu(self.dense1(X))
        X = F.softmax(self.output(X), dim=-1)
        return X

I prefer inputting lists of neurons in each layer i.e num_units=[30,15,5,2] would have 2 hidden layers with 15 and 5 neurons. Furthermore we have 30 features and 2 classes, thus re-writing it to something like this

class Net(nn.Module):
    def __init__(
            self,
            num_units=[30,15,5,2],
            nonlin=[F.relu,F.relu,F.relu],
            dropout=[0.5,0.5,0.5],
            ):
        super(Net, self).__init__()

        self.layer_units = layer_units
        self.nonlin = nonlin #Activation function
        self.dropout = dropout #Drop-out rates in each layer
        self.layers = [nn.Linear(i,p) for i,p in zip(layer_units,layer_units[1:])] #Dense layers

    def forward(self, X, **kwargs):
        print("Forwards")
        for layer,func,drop in zip(self.layers[:-1],self.nonlin,self.dropout):
            print(layer,func,drop)
            X=drop(func(layer(X)))

        X = F.softmax(X, dim=-1)
        return X

should do the trick. The problem is that when calling

net = NeuralNetClassifier(Net,max_epochs=20,lr=0.1,device="cuda")
net.fit(X,y)

I get the error "ValueError: optimizer got an empty parameter list". I have narrowed it down to the removal of self.output = nn.Linear(10, 2) simply makes the net not enter forward i.e it seems like output is some kind of "trigger" variable. Is that really the case the network need a variable called output (being a layer) at the end, and that we are not free to define the variable-names ourself ?

Output:
Pytorch will look for subclasses of nn.Module, so changing

self.layers = [nn.Linear(i,p) for i,p in zip(layer_units,layer_units[1:])]

to

self.layers = nn.ModuleList([nn.Linear(i,p) for i,p in zip(layer_units,layer_units[1:])])

should work fine


Instruction: Convert a list of tensors to tensors of tensors pytorch

Input:
I have this code:

import torch

list_of_tensors = [ torch.randn(3), torch.randn(3), torch.randn(3)]
tensor_of_tensors = torch.tensor(list_of_tensors)

I am getting the error:

  ValueError: only one element tensors can be converted to Python scalars

How can I convert the list of tensors to a tensor of tensors in pytorch?

Output:
Here is a solution:
tensor_of_tensors = torch.stack((list_of_tensors))
print(tensor_of_tensors) #shape (3,3)


Instruction: PyTorch augmentation

Input:
i'm new to machine learning and pytorch. I'm using imgaug library for images augmentation (https://github.com/aleju/imgaug)
I have this code:
class ImgAugTransform:
    def __init__(self):
        self.aug = seq = iaa.Sequential(
            [
                # Apply the following augmenters to most images
                iaa.Fliplr(0.5), # horizontally flip 50% of all images
                iaa.Flipud(0.2),  # vertically flip 20% of all images
                random_aug_use(iaa.CropAndPad( # crop images by -5% to 10% of their height/width
                    percent=(-0.1, 0.2),
                    pad_mode=ia.ALL,
                    pad_cval=(0.,255)
                )),
                random_aug_use(iaa.Affine(
                    scale={"x": (0.8, 1.2), "y": (0.8, 1.2)}, # scale images to 80-120% of their size, individually per axis
                    translate_percent={"x": (-0.2, 0.2), "y": (-0.2, 0.2)}, # translate by -20 to +20 percent (per axis)
                    rotate=(-45, 45), # rotate by -45 to +45 degrees
                    shear=(-16, 16), # shear by -16 to +16 degrees
                    order=[0, 1], # use nearest neighbour or bilinear interpolation (fast)
                    cval=(0, 255), # if mode is constant, use a cval between 0 and 255
                    mode=ia.ALL # use any of scikit-image's warping modes (see 2nd image from the top for examples)
                ))
            ],
            random_order=True)

    def __call__(self, img):
        img = np.array(img)
        return self.aug.augment_image(img)

train_transforms = ImgAugTransform()

train_dataset = torchvision.datasets.ImageFolder(train_dir, train_transforms)

train_dataloader = torch.utils.data.DataLoader(
    train_dataset, batch_size=batch_size, shuffle=True, num_workers=batch_size)

So now i cant do this:
X_batch, y_batch = next(iter(train_dataloader))

I get error:
ValueError: some of the strides of a given numpy array are negative. This is currently not supported, but will be added in future releases.

Output:
You should make your augmented numpy arrays contiguous again.
try modifying your augmenter code to:

    def __call__(self, img):
        img = np.array(img)
        return np.ascontiguousarray(self.aug.augment_image(img))


Instruction: Fit a Gaussian curve with a neural network using Pytorch

Input:
Suppose the following model :

import torch.nn as nn

class PGN(nn.Module):
    def __init__(self, input_size):
        super(PGN, self).__init__()

        self.linear = nn.Sequential(
            nn.Linear(in_features=input_size, out_features=128),
            nn.ReLU(),
            nn.Linear(in_features=128, out_features=1)
        )

    def forward(self, x):
        return self.linear(x)

I figure I have to modify the model to fit a 2-dimensional curve.

Is there a way to fit a Gaussian curve with mu=0 and sigma=0 using Pytorch? If so, can you show me?

Output:
A neural network can approximate an arbitrary function of any number of parameters to a space of any dimension.

To fit a 2 dimensional curve your network should be fed with vectors of size 2, that is a vector of x and y coordinates. The output is a single value of size 1.

For training you must generate ground truth data, that is a mapping between coordinates (x and y) and the value (z). The loss function should compare this ground truth value with the estimate of your network.

If it is just a tutorial to learn Pytorch and not a real application, you can define a function that for a given x and y output the gaussian value according to your parameters.

Then during training you randomly choose a x and y and feed this to the networks then do backprop with the true value.


Instruction: 'Sequential' object has no attribute 'classifier'

Input: How to find in_features of a Pytorch model? model.classifier.in_features is working on densenet121 but nit on vgg18, is there any function which may work on all torchvision models?

Output:
classifier is a Sequential module in the VGG's implementation, so, if you want to access the in_features passed to the classifier, you have to check the in_features of the first layer.

models.vgg19().classifier[0].in_features

It seems like different implementations follow different patterns, so the best way to determine in_features for all the models would be to check the source code directly.


Instruction: Display image in a PIL format from torch.Tensor

Input:
I’m quite new to Pytorch. I was wondering how I could convert my tensor of size torch.Size([1, 3, 224, 224]) to display in an image format on a Jupyter notebook. A PIL format or a CV2 format should be fine.

I tried using transforms.ToPILImage(x) but it resulted in a different format like this: ToPILImage(mode=ToPILImage(mode=tensor([[[[1.3034e-16, 1.3034e-16, 1.3034e-16, ..., 1.4475e-16,.

Maybe I’m doing something wrong :no_mouth:

Output:
Since your image is normalized, you need to unnormalize it. You have to do the reverse operations that you did during normalization. One way is

class UnNormalize(object):
    def __init__(self, mean, std):
        self.mean = mean
        self.std = std

    def __call__(self, tensor):
        """
        Args:
            tensor (Tensor): Tensor image of size (C, H, W) to be normalized.
        Returns:
            Tensor: Normalized image.
        """
        for t, m, s in zip(tensor, self.mean, self.std):
            t.mul_(s).add_(m)
            # The normalize code -> t.sub_(m).div_(s)
        return tensor

To use this, you'll need the mean and standard deviation (which you used to normalize the image). Then,

unorm = UnNormalize(mean = [0.35675976, 0.37380189, 0.3764753], std = [0.32064945, 0.32098866, 0.32325324])
image = unorm(normalized_image)


Instruction: Using ModuleDict, I have: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Input:
I'm trying in my __init__ function:

        self.downscale_time_conv = np.empty(8, dtype=object)
        for i in range(8):
            self.downscale_time_conv[i] = torch.nn.ModuleDict({})

But in my forward, I have:

        down_out = False
        for i in range(8):
            if not down_out:
                down_out = self.downscale_time_conv[i][side](inputs)
            else:
                down_out += self.downscale_time_conv[i][side](inputs)

and I get:

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Output:
        self.downscale_time_conv = torch.nn.ModuleList()
        for i in range(8):
            self.downscale_time_conv.append(torch.nn.ModuleDict({}))

this solved it. Apparently I needed to use a ModuleList