Instruction: Have I implemented implemenation of learning rate finder correctly?

Input:
Using implementation of lr_finder from https://github.com/davidtvs/pytorch-lr-finder based on paper https://arxiv.org/abs/1506.01186

Without the learning rate finder :

from __future__ import print_function, with_statement, division
import torch
from tqdm.autonotebook import tqdm
from torch.optim.lr_scheduler import _LRScheduler
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
import torchvision.datasets as dset
import os
import torch.nn.functional as F
import time
import random
import pickle
from sklearn.metrics import confusion_matrix
import pandas as pd
import sklearn

class LRFinder(object):
    """Learning rate range test.

    The learning rate range test increases the learning rate in a pre-training run
    between two boundaries in a linear or exponential manner. It provides valuable
    information on how well the network can be trained over a range of learning rates
    and what is the optimal learning rate.

    Arguments:
        model (torch.nn.Module): wrapped model.
        optimizer (torch.optim.Optimizer): wrapped optimizer where the defined learning
            is assumed to be the lower boundary of the range test.
        criterion (torch.nn.Module): wrapped loss function.
        device (str or torch.device, optional): a string ("cpu" or "cuda") with an
            optional ordinal for the device type (e.g. "cuda:X", where is the ordinal).
            Alternatively, can be an object representing the device on which the
            computation will take place. Default: None, uses the same device as `model`.

    Example:
        >>> lr_finder = LRFinder(net, optimizer, criterion, device="cuda")
        >>> lr_finder.range_test(dataloader, end_lr=100, num_iter=100)

    Cyclical Learning Rates for Training Neural Networks: https://arxiv.org/abs/1506.01186
    fastai/lr_find: https://github.com/fastai/fastai

    """

    def __init__(self, model, optimizer, criterion, device=None):
        self.model = model
        self.optimizer = optimizer
        self.criterion = criterion
        self.history = {"lr": [], "loss": []}
        self.best_loss = None

        # Save the original state of the model and optimizer so they can be restored if
        # needed
        self.model_state = model.state_dict()
        self.model_device = next(self.model.parameters()).device
        self.optimizer_state = optimizer.state_dict()

        # If device is None, use the same as the model
        if device:
            self.device = device
        else:
            self.device = self.model_device

    def reset(self):
        """Restores the model and optimizer to their initial states."""
        self.model.load_state_dict(self.model_state)
        self.model.to(self.model_device)
        self.optimizer.load_state_dict(self.optimizer_state)

    def range_test(
        self,
        train_loader,
        val_loader=None,
        end_lr=10,
        num_iter=100,
        step_mode="exp",
        smooth_f=0.05,
        diverge_th=5,
    ):
        """Performs the learning rate range test.

        Arguments:
            train_loader (torch.utils.data.DataLoader): the training set data laoder.
            val_loader (torch.utils.data.DataLoader, optional): if `None` the range test
                will only use the training loss. When given a data loader, the model is
                evaluated after each iteration on that dataset and the evaluation loss
                is used. Note that in this mode the test takes significantly longer but
                generally produces more precise results. Default: None.
            end_lr (float, optional): the maximum learning rate to test. Default: 10.
            num_iter (int, optional): the number of iterations over which the test
                occurs. Default: 100.
            step_mode (str, optional): one of the available learning rate policies,
                linear or exponential ("linear", "exp"). Default: "exp".
            smooth_f (float, optional): the loss smoothing factor within the [0, 1[
                interval. Disabled if set to 0, otherwise the loss is smoothed using
                exponential smoothing. Default: 0.05.
            diverge_th (int, optional): the test is stopped when the loss surpasses the
                threshold:  diverge_th * best_loss. Default: 5.

        """
        # Reset test results
        self.history = {"lr": [], "loss": []}
        self.best_loss = None

        # Move the model to the proper device
        self.model.to(self.device)

        # Initialize the proper learning rate policy
        if step_mode.lower() == "exp":
            lr_schedule = ExponentialLR(self.optimizer, end_lr, num_iter)
        elif step_mode.lower() == "linear":
            lr_schedule = LinearLR(self.optimizer, end_lr, num_iter)
        else:
            raise ValueError("expected one of (exp, linear), got {}".format(step_mode))

        if smooth_f < 0 or smooth_f >= 1:
            raise ValueError("smooth_f is outside the range [0, 1[")

        # Create an iterator to get data batch by batch
        iterator = iter(train_loader)
        for iteration in tqdm(range(num_iter)):
            # Get a new set of inputs and labels
            try:
                inputs, labels = next(iterator)
            except StopIteration:
                iterator = iter(train_loader)
                inputs, labels = next(iterator)

            # Train on batch and retrieve loss
            loss = self._train_batch(inputs, labels)
            if val_loader:
                loss = self._validate(val_loader)

            # Update the learning rate
            lr_schedule.step()
            self.history["lr"].append(lr_schedule.get_lr()[0])

            # Track the best loss and smooth it if smooth_f is specified
            if iteration == 0:
                self.best_loss = loss
            else:
                if smooth_f > 0:
                    loss = smooth_f * loss + (1 - smooth_f) * self.history["loss"][-1]
                if loss < self.best_loss:
                    self.best_loss = loss

            # Check if the loss has diverged; if it has, stop the test
            self.history["loss"].append(loss)
            if loss > diverge_th * self.best_loss:
                print("Stopping early, the loss has diverged")
                break

        print("Learning rate search finished. See the graph with {finder_name}.plot()")

    def _train_batch(self, inputs, labels):
        # Set model to training mode
#         self.model.train()

        # Move data to the correct device
        inputs = inputs.to(self.device)
        labels = labels.to(self.device)

        # Forward pass
        self.optimizer.zero_grad()
        outputs = self.model(inputs)
        loss = self.criterion(outputs, labels)

        # Backward pass
        loss.backward()
        self.optimizer.step()

        return loss.item()

    def _validate(self, dataloader):
        # Set model to evaluation mode and disable gradient computation
        running_loss = 0
        self.model.eval()
        with torch.no_grad():
            for inputs, labels in dataloader:
                # Move data to the correct device
                inputs = inputs.to(self.device)
                labels = labels.to(self.device)

                # Forward pass and loss computation
                outputs = self.model(inputs)
                loss = self.criterion(outputs, labels)
                running_loss += loss.item() * inputs.size(0)

        return running_loss / len(dataloader.dataset)

    def plot(self, skip_start=10, skip_end=5, log_lr=True):
        """Plots the learning rate range test.

        Arguments:
            skip_start (int, optional): number of batches to trim from the start.
                Default: 10.
            skip_end (int, optional): number of batches to trim from the start.
                Default: 5.
            log_lr (bool, optional): True to plot the learning rate in a logarithmic
                scale; otherwise, plotted in a linear scale. Default: True.

        """

        if skip_start < 0:
            raise ValueError("skip_start cannot be negative")
        if skip_end < 0:
            raise ValueError("skip_end cannot be negative")

        # Get the data to plot from the history dictionary. Also, handle skip_end=0
        # properly so the behaviour is the expected
        lrs = self.history["lr"]
        losses = self.history["loss"]
        if skip_end == 0:
            lrs = lrs[skip_start:]
            losses = losses[skip_start:]
        else:
            lrs = lrs[skip_start:-skip_end]
            losses = losses[skip_start:-skip_end]

        # Plot loss as a function of the learning rate
        plt.plot(lrs, losses)
        if log_lr:
            plt.xscale("log")
        plt.xlabel("Learning rate")
        plt.ylabel("Loss")
        plt.show()

class LinearLR(_LRScheduler):
    """Linearly increases the learning rate between two boundaries over a number of
    iterations.

    Arguments:
        optimizer (torch.optim.Optimizer): wrapped optimizer.
        end_lr (float, optional): the initial learning rate which is the lower
            boundary of the test. Default: 10.
        num_iter (int, optional): the number of iterations over which the test
            occurs. Default: 100.
        last_epoch (int): the index of last epoch. Default: -1.

    """

    def __init__(self, optimizer, end_lr, num_iter, last_epoch=-1):
        self.end_lr = end_lr
        self.num_iter = num_iter
        super(LinearLR, self).__init__(optimizer, last_epoch)

    def get_lr(self):
        curr_iter = self.last_epoch + 1
        r = curr_iter / self.num_iter
        return [base_lr + r * (self.end_lr - base_lr) for base_lr in self.base_lrs]

class ExponentialLR(_LRScheduler):
    """Exponentially increases the learning rate between two boundaries over a number of
    iterations.

    Arguments:
        optimizer (torch.optim.Optimizer): wrapped optimizer.
        end_lr (float, optional): the initial learning rate which is the lower
            boundary of the test. Default: 10.
        num_iter (int, optional): the number of iterations over which the test
            occurs. Default: 100.
        last_epoch (int): the index of last epoch. Default: -1.

    """

    def __init__(self, optimizer, end_lr, num_iter, last_epoch=-1):
        self.end_lr = end_lr
        self.num_iter = num_iter
        super(ExponentialLR, self).__init__(optimizer, last_epoch)

    def get_lr(self):
        curr_iter = self.last_epoch + 1
        r = curr_iter / self.num_iter
        return [base_lr * (self.end_lr / base_lr) ** r for base_lr in self.base_lrs]

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

root = './data'
if not os.path.exists(root):
    os.mkdir(root)
train_set = dset.MNIST(root=root, train=True, transform=trans, download=True)
test_set = dset.MNIST(root=root, train=False, transform=trans, download=True)

batch_size = 64

train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True)

test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
shuffle=True)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 10)
    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

num_epochs = 2
random_sample_size = 200

# Hyper-parameters
input_size = 100
hidden_size = 100
num_classes = 10
learning_rate = .0001
# Device configuration
device = 'cpu'

model = NeuralNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
# lr_finder = LRFinder(model, optimizer, criterion, device="cpu")
# lr_finder.range_test(train_loader, end_lr=100, num_iter=100)
# lr_finder.plot()
# optimizer = torch.optim.Adam(model.parameters(), lr=lr_finder.history['lr'][0])
# print(lr_finder.history['lr'])

predicted_test = []
labels_l = []
actual_values = []
predicted_values = []

N = len(train_loader)
# Train the model
total_step = len(train_loader)

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
#         images = images.reshape(-1, 50176).to(device)
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        predicted = outputs.data.max(1)[1]
        predicted_test.append(predicted.cpu().numpy())
        labels_l.append(labels.cpu().numpy())

        loss = criterion(outputs, labels)
        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    predicted_values.append(np.concatenate(predicted_test).ravel())
    actual_values.append(np.concatenate(labels_l).ravel())

    print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))
    print('training accuracy : ', 100 * len((np.where(np.array(predicted_values[0])==(np.array(actual_values[0])))[0])) / len(actual_values[0]))

Results :

Epoch [1/2], Step [938/938], Loss: 0.5374
training accuracy :  84.09833333333333
Epoch [2/2], Step [938/938], Loss: 0.2055
training accuracy :  84.09833333333333

With the learning rate finder code being uncommented :

Below commented out code is now not un-commented :

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
lr_finder = LRFinder(model, optimizer, criterion, device="cpu")
lr_finder.range_test(train_loader, end_lr=100, num_iter=100)
lr_finder.plot()
optimizer = torch.optim.Adam(model.parameters(), lr=lr_finder.history['lr'][0])
print(lr_finder.history['lr'])

the model achieves results after two epochs :

Epoch [1/2], Step [938/938], Loss: 3.7311
training accuracy :  9.93
Epoch [2/2], Step [938/938], Loss: 3.5106
training accuracy :  9.93

Can see the training accuracy is much lower 84.09833333333333 versus 9.93 . Should the learning rate finder find a learning rate that allows to achieve greater training set accuracy ?

Output:
The code looks like it's using the implementation correctly. To answer your last question,

  Can see the training accuracy is much lower 84.09833333333333 versus 9.93 . Should the learning rate finder find a learning rate that allows to achieve greater training set accuracy ?

Not really. A few points

You are using Adam, which scales the learning rate adaptively for each parameter in the network. The initial learning rate will matter less, as opposed to traditional SGD, for example. The original authors of Adam write

  The hyper-parameters have intuitive interpre-tations and typically require little tuning. [1]

A well tuned learning rate should make your network converge faster (i.e in less epochs). It might still find the same local minima as a higher learning rate, but faster. The risk with too high learning rates is that you overshoot the local minima and instead find a poor one. With a tiny learning rate you should get the best training accuracy, but it will take very long.
You are training your model for only 2 epochs. If I had to guess, the algorithm has found that a small learning rate leads to good optima, but since it is small, it requires more time to converge. To test this theory, I would recommend running your training longer.

All that said, your time is probably better spent using Adam with default parameters and directing your attention elsewhere, such as modelling choices (layers, nodes, activations, etc). In my experience standard Adam works really well in most cases.

[1] https://arxiv.org/abs/1412.6980


Instruction: LSTM's expected hidden state dimensions doesn't take batch size into account

Input:
I have this decoder model, which is supposed to take batches of sentence embeddings (batchsize = 50, hidden size=300) as input and output a batch of one hot representation of predicted sentences:

class DecoderLSTMwithBatchSupport(nn.Module):
        # Your code goes here
        def __init__(self, embedding_size,batch_size, hidden_size, output_size):
            super(DecoderLSTMwithBatchSupport, self).__init__()
            self.hidden_size = hidden_size
            self.batch_size = batch_size
            self.lstm = nn.LSTM(input_size=embedding_size,num_layers=1, hidden_size=hidden_size, batch_first=True)
            self.out = nn.Linear(hidden_size, output_size)
            self.softmax = nn.LogSoftmax(dim=1)

        def forward(self, my_input, hidden):
            print(type(my_input), type(hidden))
            output, hidden = self.lstm(my_input, hidden)
            output = self.softmax(self.out(output[0]))
            return output, hidden

        def initHidden(self):
            return Variable(torch.zeros(1, self.batch_size, self.hidden_size)).cuda()

However, when I run it using:

decoder=DecoderLSTMwithBatchSupport(vocabularySize,batch_size, 300, vocabularySize)
decoder.cuda()
decoder_input=np.zeros([batch_size,vocabularySize])
    for i in range(batch_size):
        decoder_input[i] = embeddings[SOS_token]
    decoder_input=Variable(torch.from_numpy(decoder_input)).cuda()
    decoder_hidden = (decoder.initHidden(),decoder.initHidden())
        for di in range(target_length):
            decoder_output, decoder_hidden = decoder(decoder_input.view(1,batch_size,-1), decoder_hidden)

I get he following error:

  Expected hidden[0] size (1, 1, 300), got (1, 50, 300)

What am I missing in order to make the model expect batched hidden states?

Output:
When you create the LSTM, the flag batch_first is not necessary, because it assumes a different shape of your input. From the docs:

  If True, then the input and output tensors are provided as (batch,
  seq, feature). Default: False

change the LSTM creation to:

self.lstm = nn.LSTM(input_size=embedding_size, num_layers=1, hidden_size=hidden_size)

Also, there is a type error. When you create the decoder_input using torch.from_numpy() it has a dtype=torch.float64, while decoder_input has as default dtype=torch.float32. Change the line where you create the decoder_input to something like

decoder_input = Variable(torch.from_numpy(decoder_input)).cuda().float()

With both changes, it is supposed to work fine :)


Instruction: Result of auto-encoder dimensions are incorrect

Input:
Using below code I'm attempting to encode image from mnist into a lower dimension representation :

import warnings
warnings.filterwarnings('ignore')
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from matplotlib import pyplot as plt
from sklearn import metrics
import datetime
from sklearn.preprocessing import MultiLabelBinarizer
import seaborn as sns
sns.set_style("darkgrid")
from ast import literal_eval
import numpy as np
from sklearn.preprocessing import scale
import seaborn as sns
sns.set_style("darkgrid")
import torch
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

%matplotlib inline

low_dim_rep = 32
epochs = 2

cuda = torch.cuda.is_available() # True if cuda is available, False otherwise
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
print('Training on %s' % ('GPU' if cuda else 'CPU'))

# Loading the MNIST data set
transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),
                torchvision.transforms.Normalize((0.1307,), (0.3081,))])
mnist = torchvision.datasets.MNIST(root='../data/', train=True, transform=transform, download=True)

# Loader to feed the data batch by batch during training.
batch = 100
data_loader = torch.utils.data.DataLoader(mnist, batch_size=batch, shuffle=True)

encoder = nn.Sequential(
                # Encoder
                nn.Linear(28 * 28, 64),
                nn.PReLU(64),
                nn.BatchNorm1d(64),

                # Low-dimensional representation
                nn.Linear(64, low_dim_rep),
                nn.PReLU(low_dim_rep),
                nn.BatchNorm1d(low_dim_rep))

decoder = nn.Sequential(
                # Decoder
                nn.Linear(low_dim_rep, 64),
                nn.PReLU(64),
                nn.BatchNorm1d(64),
                nn.Linear(64, 28 * 28))

autoencoder = nn.Sequential(encoder, decoder)

encoder = encoder.type(FloatTensor)
decoder = decoder.type(FloatTensor)
autoencoder = autoencoder.type(FloatTensor)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.00001)

data_size = int(mnist.train_labels.size()[0])

print('data_size' , data_size)
for i in range(epochs):
    for j, (images, _) in enumerate(data_loader):
        images = images.view(images.size(0), -1) # from (batch 1, 28, 28) to (batch, 28, 28)
        images = Variable(images).type(FloatTensor)

        autoencoder.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()
    print('Epoch %i/%i loss %.2f' % (i + 1, epochs, loss.data[0]))

print('Optimization finished.')

# Get the encoded images here
encoded_images = []
for j, (images, _) in enumerate(data_loader):
    images = images.view(images.size(0), -1)
    images = Variable(images).type(FloatTensor)

    encoded_images.append(encoder(images))

Upon completion of this code

len(encoded_images) is 600 when I expect the length to match the number of images in mnist : len(mnist) - 60'000.

How to encode the images to a lower dimension representation of 32 ( low_dim_rep = 32 ) ? I've defined the network parameters incorrectly ?

Output:
You have 60000 images in mnist and your batch = 100. That is why your len(encoded_images)=600 because you do 60000/100=600 iterations  when generating encoded image. You end up with a list of 600 elements where each element has shape [100, 32]. You can do the following

encoded_images = torch.zeros(len(mnist), 32)
for j, (images, _) in enumerate(data_loader):
    images = images.view(images.size(0), -1)
    images = Variable(images).type(FloatTensor)
    encoded_images[j * batch : (j+1) * batch] = encoder(images)


Instruction: Store weight updates for momemntum

Input:
I am trying to implement momentum in my implementation of SGD with momentum.
From my understanding this update look like this:

parameters -= (lr * (p.grad*0.1 + p_delta_prev*0.9))

My question is how I should store my previous deltas from every update

Here is what I have in my update function:

#we now want to do the update with momentum
#momentum takes derivative, multiplies it by 0.1, then takes the previous update,
#multiplies it by 0.9 and we add the two together
#alpha = 0.1, beta = 0.9;  p-=grad*0.1 + p*0.9
def update(x,y,lr):
    wd = 1e-5
    y_hat = model(x)
    # weight decay
    w2 = 0.
    for p in model.parameters(): w2 += (p**2).sum()
    # add to regular loss
    loss = loss_func(y_hat, y) + w2*wd
    loss.backward()
    with torch.no_grad():
        for p in model.parameters():
            #p.grad is the slope of the line of that parameter
            #current_p-previous_p to get difference
            p_update = (lr * (p.grad*0.1 + p*0.9))
            p.sub_(p_update)
            p.grad.zero_()
    return loss.item()

Here the p*0.9 should be replace by the p_delta_prev. But how should I store these deltas for every parameter? If I save them to a tensor wouldn't I would be effectively copying the weight deltas to memory making my model two times the size. What would be a good way to accomplish this? I do not want to use an inbuilt function that does that activation for me. I did look into the pytorch sgd.py and it looks like the store the states.

I have updated the code:

#we now want to do the update with momentum
#momentum takes derivative, multiplys it by 0.1, then takes the previous update,
#multiplies it by 0.9 and we add the two together
#alpha = 0.1, beta = 0.9;  p-=grad*0.1 + p*0.9
p_delta = {}
def update(x,y,lr):
    wd = 1e-5
    y_hat = model(x)
    # weight decay
    w2 = 0.
    for p in model.parameters(): w2 += (p**2).sum()
    # add to regular loss
    loss = loss_func(y_hat, y) + w2*wd
    loss.backward()
    with torch.no_grad():
        i = 0
        for p in model.parameters():
            #p.grad is the slope of the line of that parameter
            if i not in p_delta:#check if key exists
                p_delta[i] = torch.zeros_like(p)
            p_update = (lr *p.grad) + (p_delta[i]*0.9)
            p_delta[i] = p_update.clone()
            p.sub_(p_update)
            p.grad.zero_()
            print((p_delta[i]))
            i+=1
    return loss.item()

I think the code in the Excel spreadsheet is incorrect. Jeremy seems to show: lr* ((p.grad*0.1) + (p_delta[i]*0.9)) but many tutorials seem to show: (lr *p.grad) + (p_delta[i]*0.9) If we implement Jeremy’s code the loss actually is slower than vanilla GD. The part of the video is here: https://youtu.be/CJKnDu2dxOE?t=6581

Output:
Yes, it does store the parameter momenta in a dictionary, indexed by their names, as returned by model.named_parameters(). I don't know how to rigorously prove this, but I strongly believe it's impossible to apply momentum without using additional memory twice the size of your model.

That being said, I wouldn't worry, because model size is rarely a big factor in memory consumption of the whole algorithm - keeping the intermediate network activations for backpropagation algorithm is far more expensive. Taking the VGG-16 network as an example, it has 138 million parameters (figure taken from here), which amounts to slightly more than 0.5gb if stored in single precision. Compare this with 6gb+ found on reasonable modern GPUs.


Instruction: How to reset Colab after the following CUDA error 'Cuda assert fails: device-side assert triggered'?

Input:
I'm running my Jupyter Notebook using Pytorch on Google Colab. After I received the 'Cuda assert fails: device-side assert triggered' I am unable to run any other code that uses my pytorch module. Does anyone know how to reset my code so that my Pytorch functions that were working before can still run?

I've already tried implementing CUDA_LAUNCH_BLOCKING=1but my code still doesn't work as the Assert is still triggered!

Output:
You need to reset the Colab notebook. To run existing Pytorch modules that used to work before, you have to do the following:

Go to 'Runtime' in the tool bar
Click 'Restart and Run all'

This will reset your CUDA assert and flush out the module so that you can have another shot at avoiding the error!


Instruction: How to wrap PyTorch functions and implement autograd?

Input:
I'm working through the PyTorch tutorial on Defining new autograd functions. The autograd function I want to implement is a wrapper around torch.nn.functional.max_pool1d. Here is what I have so far:

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.autograd as tag

class SquareAndMaxPool1d(tag.Function):

    @staticmethod
    def forward(ctx, input, kernel_size, stride=None, padding=0, dilation=1, \
                return_indices=False, ceil_mode=False):
        ctx.save_for_backward( input )

        inputC = input.clone() #copy input
        inputC *= inputC

        output = F.max_pool1d(inputC, kernel_size, stride=stride, \
                              padding=padding, dilation=dilation, \
                              return_indices=return_indices, \
                              ceil_mode=ceil_mode)

        return output

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = get_max_pool1d_grad_somehow(grad_output)
        return 2.0*input*grad_input

My question is: how to I get the gradient of the wrapped function? I know that there are probably other ways to do this given how simple the example I present is, but what I want to do fits this framework and requires me to implement an autograd function.

Edit: After examining this blog post I decided to try the following for backward:

def backward(ctx, grad_output):
    input, output = ctx.saved_tensors
    grad_input = output.backward(grad_output)
    return 2.0*input*grad_input

with output added to the saved variables. I then run the following code:

x = np.random.randn(1,1,5)
xT = torch.from_numpy(x)
xT.requires_grad=True
f = SquareAndMaxPool1d.apply
s = torch.sum(f(xT,2))
s.backward()

and I get Bus error: 10.

Say, xT is tensor([[[ 1.69533562, -0.21779421,  2.28693953, -0.86688095, -1.01033497]]], dtype=torch.float64), then I would expect to find that xT.grad is  tensor([[[ 3.39067124, -0.        ,  9.14775812, -0.        , -2.02066994]]], dtype=torch.float64) after calling s.backward() (that is 2*x*grad_of_max_pool, with grad_of_max_pool containing tensor([[[1., 0., 2., 0., 1.]]], dtype=torch.float64)).

I've figured out why I get a Bus error: 10. It appears that the above code leads to a recursive call of my backward at grad_input = output.backward(grad_output). So I need to find some other way to get the gradient of max_pool1d. I know how to implement this in pure Python, but the result would be much slower than if I could wrap the library code.

Output:
You have picked a rather unlucky example. torch.nn.functional.max_pool1d is not an instance of torch.autograd.Function, because it's a PyTorch built-in, defined in C++ code and with an autogenerated Python binding. I am not sure if it's possible to get the backward property via its interface.

Firstly, in case you haven't noticed, you don't need to write any custom code for backpropagation of this formula because both power operation and max_pool1d already have it defined, so their composition also is covered by the autograd. Assuming your goal is an exercise, I would suggest you do it more manually (without falling back to backward of max_pool1d). An example is below

import torch
import torch.nn.functional as F
import torch.autograd as tag

class SquareAndMaxPool1d(tag.Function):
    @staticmethod
    def forward(ctx, input, kernel_size, **kwargs):
        # we're gonna need indices for backward. Currently SquareAnd...
        # never actually returns indices, I left it out for simplicity
        kwargs['return_indices'] = True

        input_sqr = input ** 2
        output, indices = F.max_pool1d(input_sqr, kernel_size, **kwargs)
        ctx.save_for_backward(input, indices)

        return output

    @staticmethod
    def backward(ctx, grad_output):
        input, indices = ctx.saved_tensors

        # first we need to reconstruct the gradient of `max_pool1d`
        # by putting all the output gradient elements (corresponding to
        # input elements which made it through the max_pool1d) in their
        # respective places, the rest has gradient of 0. We do it by
        # scattering it against a tensor of 0s
        grad_output_unpooled = torch.zeros_like(input)
        grad_output_unpooled.scatter_(2, indices, grad_output)

        # then incorporate the gradient of the "square" part of your
        # operator
        grad_input = 2. * input * grad_output_unpooled

        # the docs for backward
        # https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function.backward
        # say that "it should return as many tensors, as there were inputs
        # to forward()". It fails to mention that if an argument was not a
        # tensor, it should return None (I remember reading this somewhere,
        # but can't find it anymore). Anyway, we need to
        # return a (grad_input, None) tuple to avoid a complaint that two
        # outputs were expected
        return grad_input, None

We can then use the numerical gradient checker to verify that the operation works as expected.

f = SquareAndMaxPool1d.apply
xT = torch.randn(1, 1, 6, requires_grad=True, dtype=torch.float64)
tag.gradcheck(lambda t: f(t, 2), xT)

I'm sorry if this doesn't address your question of how to get the backward of max_pool1d, but hopefully you find my answer useful enough.


Instruction: Pytorch equivalent of Numpy's logical_and and kin?

Input:
Does Pytorch have an equivalent of Numpy's element-wise logical operators (logical_and, logical_or, logical_not, and logical_xor)? Calling the Numpy functions on Pytorch tensors seems to work well enough when using the CPU, even producing a Pytorch tensor as output. I mainly ask because I assume this would not work so well if the pytorch calculation were running in the GPU.

I've looked through Pytorch's documentation index at all functions containing the string "and" and none seem relevant.

Output:
Update: With Pytorch 1.2, PyTorch introduced torch.bool datatype, which can be used using torch.BoolTensor:

>>> a = torch.BoolTensor([False, True, True, False])  # or pass [0, 1, 1, 0]
>>> b = torch.BoolTensor([True, True, False, False])

>>> a & b  # logical and
tensor([False,  True, False, False])

PyTorch supports logical operations on ByteTensor. You can use logical operations using &, |, ^, ~ operators as follows:

>>> a = torch.ByteTensor([0, 1, 1, 0])
>>> b = torch.ByteTensor([1, 1, 0, 0])

>>> a & b  # logical and
tensor([0, 1, 0, 0], dtype=torch.uint8)

>>> a | b  # logical or
tensor([1, 1, 1, 0], dtype=torch.uint8)

>>> a ^ b  # logical xor
tensor([1, 0, 1, 0], dtype=torch.uint8)

>>> ~a  # logical not
tensor([1, 0, 0, 1], dtype=torch.uint8)


Instruction: Why/how can model.forward() succeed both on input being mini-batch vs single item?

Input:
Why and how does this work?

When I run the forward phase on input

being mini-batch tensor
or alternatively being a single input item

model.__call__() (which AFAIK is calling forward() ) swallows that and spills out adequate output (i.e. a tensor of mini-batch of estimates or a single item of estimate)

Adopting testcode from the Pytorch NN example shows what I mean, but I don't get it.

I would expect it to create problems and me forced to transform the single item input into a mini-batch of size 1( reshape (1,xxx)) or likewise, like I did in the code below.

( I did variations of the test to be sure it is e.g. not depending on execution order )

# -*- coding: utf-8 -*-
import torch

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
#N, D_in, H, D_out = 64, 1000, 100, 10
N, D_in, H, D_out = 64, 10, 4, 3

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

# Use the nn package to define our model as a sequence of layers. nn.Sequential
# is a Module which contains other Modules, and applies them in sequence to
# produce its output. Each Linear Module computes output from input using a
# linear function, and holds internal Tensors for its weight and bias.
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

# The nn package also contains definitions of popular loss functions; in this
# case we will use Mean Squared Error (MSE) as our loss function.
loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-4
for t in range(1):
    # Forward pass: compute predicted y by passing x to the model. Module objects
    # override the __call__ operator so you can call them like functions. When
    # doing so you pass a Tensor of input data to the Module and it produces
    # a Tensor of output data.
    model.eval()
    print ("###########")
    print ("x[0]",x[0])
    print ("x[0].size()", x[0].size())
    y_1pred = model(x[0])
    print ("y_1pred.size()", y_1pred.size())
    print (y_1pred)

    model.eval()
    print ("###########")
    print ("x.size()", x.size())
    y_pred = model(x)
    print ("y_pred.size()", y_pred.size())
    print ("y_pred[0]", y_pred[0])

    print ("###########")
    model.eval()
    input_item = x[0]
    batch_len1_shape = torch.Size([1,*(input_item.size())])
    batch_len1 = input_item.reshape(batch_len1_shape)
    y_pred_batch_len1 = model(batch_len1)

    print ("input_item",input_item)
    print ("input_item.size()", input_item.size())
    print ("y_pred_batch_len1.size()", y_pred_batch_len1.size())
    print (y_1pred)

    raise Exception

This is the output it generates:

###########
x[0] tensor([-1.3901, -0.2659,  0.4352, -0.6890,  0.1098, -0.3124,  0.6419,  1.1004,
        -0.7910, -0.5389])
x[0].size() torch.Size([10])
y_1pred.size() torch.Size([3])
tensor([-0.5366, -0.4826,  0.0538], grad_fn=<AddBackward0>)
###########
x.size() torch.Size([64, 10])
y_pred.size() torch.Size([64, 3])
y_pred[0] tensor([-0.5366, -0.4826,  0.0538], grad_fn=<SelectBackward>)
###########
input_item tensor([-1.3901, -0.2659,  0.4352, -0.6890,  0.1098, -0.3124,  0.6419,  1.1004,
        -0.7910, -0.5389])
input_item.size() torch.Size([10])
y_pred_batch_len1.size() torch.Size([1, 3])
tensor([-0.5366, -0.4826,  0.0538], grad_fn=<AddBackward0>)

Output:
The docs on nn.Linear state that

  Input: (N,∗,in_features) where ∗ means any number of additional dimensions

so one would naturally expect that at least two dimensions are necessary. However, if we look under the hood we will see that Linear is implemented in terms of nn.functional.linear, which dispatches to torch.addmm or torch.matmul (depending whether bias == True) which broadcast their argument.

So this behavior is likely a bug (or an error in documentation) and I would not depend on it working in the future, if I were you.


Instruction: When to use learning rate finder

Input:
Reading the paper '
Cyclical Learning Rates for Training Neural Networks' https://arxiv.org/abs/1506.01186

Does it make sense to use the learning rate finder if the model is over-fitting ? Other than reduce the number of iterations before the model overfit's will using the learning finder prevent over-fitting ?

From reading the paper there is no suggestion this method of reduces over-fitting, is my interpretation correct ?

Output: I don't think changing the learning rate reduces over-fit. To avoid over-fitting you might want to use L1/L2 regularization and drop-out or some of its variant.


Instruction: "RuntimeError: Found 0 files in subfolders of ".. Error about subfolder in Pytorch

Input:
I'm based on Window 10, Jupyter Notebook, Pytorch 1.0, Python 3.6.x currently.

At first I confirm to the correct path of files using this code : print(os.listdir('./Dataset/images/')).

and I could check that this path is correct.

but I met Error :

  RuntimeError: Found 0 files in subfolders of: ./Dataset/images/
  Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif"

What is the matter?
Could you suggest a solution?

I tried to ./dataset/1/images like this method. but the result was same....

img_dir = './Dataset/images/'
img_data = torchvision.datasets.ImageFolder(os.path.join(img_dir), transforms.Compose([
            transforms.Scale(256),
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            ]))
img_batch = data.DataLoader(img_data, batch_size=batch_size,
                               shuffle = True, drop_last=True)

Output:
Can you post the structure of your files? In your case, it is supposed to be:

img_dir
|_class1
  |_a.jpg
  |_b.jpg
|_class2
  |_a.jpg
  |_b.jpg
...


Instruction: simple CNN model training using CIFAR-10 dataset STUCK at a low accuracy

Input:
Hi i have just learned to implement NN models in pytorch through the udacity course and thus created a simple model with a a few CNN and FC layers. after much struggle i got the model to work. but it seems that it is stuck at the same loss even after repeated executions. i dont know where i am going wrong. Must be some logical error which i cant see.
Here is the code.

model

class cifar_clasify(nn.Module):

    def __init__(self):

        super().__init__()

        self.conv1 = nn.Conv2d(3,16,3)
        self.BNorm1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16,32,3)
        self.BNorm2 = nn.BatchNorm2d(32)
        self.fc1 = nn.Linear(32*6*6,256)
        self.fc2 = nn.Linear(256,512)
        self.fc3 = nn.Linear(512,10)

        self.drop = nn.Dropout(p =0.2)

    def forward(self,x):

        out = self.conv1(x)
        out = F.relu(out)
        #print(out.shape)
        out = F.max_pool2d(out,2)
        out = self.BNorm1(out)
        #print(out.shape)

        out = self.conv2(out)
        out = F.relu(out)
        #print(out.shape)
        out = F.max_pool2d(out,2)
        out = self.BNorm2(out)
        #print(out.shape)
        out = out.view(out.shape[0],-1)

        out = self.fc1(out)
        out = self.drop(F.relu(out))
        out = self.fc2(out)
        out = self.drop(F.relu(out))
        final = F.log_softmax(F.relu(self.fc3(out)) , dim = 1)

        return final

training code

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

model = cifar_clasify()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr =0.03)

epoch =2
step = 2
running_loss = 0
accuracy = 0
print_every = 5

model.to(device)

for e in range(epoch):

    for inputs,label_ in zip(train_X,train_labels):

        step +=1
        inputs = inputs.view((-1,3,32,32))
        inputs,label_ = inputs.to(device) , label_.to(device)

        #inputs.cuda()
        #label.cuda()

        optimizer.zero_grad()

        logps = model.forward(inputs)

        loss = criterion(logps , label_.reshape(1))

        loss.backward()

        optimizer.step()

        running_loss += loss.item()

        if step % print_every == 0:
            test_loss = 0
            accuracy = 0
            model.eval()
            with torch.no_grad():
                for testx, lab in zip(test_X , test_labels):

                    testx = testx.view((-1,3,32,32))

                    testx,lab = testx.to(device) , lab.to(device)
                    lab = lab.reshape(1)
                    logps = model.forward(testx)
                    batch_loss = criterion(logps , lab)
                    #print(batch_loss.item())
                    test_loss += batch_loss.item()

                    ps = torch.exp(logps)

                    top_p , topclass = ps.topk(1,dim = 1)

                    equals = topclass == lab.view(*topclass.shape)

                    accuracy += torch.mean(torch.mean(equals.type(torch.FloatTensor))).item()
            print(f"Epoch {e+1}/{epoch}.. "
                  f"Train loss: {running_loss/print_every:.3f}.. "
                  f"Test loss: {test_loss/len(test_X):.3f}.. "
                  f"Test accuracy: {accuracy/len(test_X):.3f}")
            running_loss = 0
            model.train()

Here is the result which i had to stop as it was not improving :

Epoch 1/2.. Train loss: 1.396.. Test loss: 5.288.. Test accuracy: 0.104
step =  5
Epoch 1/2.. Train loss: 3.038.. Test loss: 2.303.. Test accuracy: 0.104
step =  10
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  15
Epoch 1/2.. Train loss: 2.669.. Test loss: 2.318.. Test accuracy: 0.105
step =  20
Epoch 1/2.. Train loss: 3.652.. Test loss: 2.303.. Test accuracy: 0.104
step =  25
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  30
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  35
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  40
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  45
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  50
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  55

Here is the code if you want any other information:

Simple CNN for CIFAR 10 classification in google colab

Output:
Since batch size is 1, use a lower learning rate like 1e-4 or increase the batch size.

I recommend making batch size 16 or larger though.

EDIT: To create a batch of data you can do something like this.

N = input.shape[0] #know the total size/samples in input
for i in range(n_epochs):
    # this is to shuffle data
    indices = torch.randperm(N)
    for idx in range(0, N, batch_size):
        batch_input = input[idx:idx+batch_size]   # this will get you input of size batch_size
        # do whatever you want with the batch_input
        # ....


Instruction: Using a generator with pickled data in a Dataloader for PyTorch

Input:
I have done some preprocessing and feature selection before, and I have a pickle training input data that consists of list of lists, e.g. (but pickled)

[[1,5,45,13], [23,256,4,2], [1,12,88,78], [-1]]
[[12,45,77,325], [23,257,5,28], [3,7,48,178], [12,77,89,99]]
[[13,22,78,89], [12,33,97], [-1], [-1]]

[-1] is a padding token, but I don't think that matters.

Because the file is numerous of gigabytes, I wish to spare memory and use a generator to read in the pickle line by line (list by list). I already found this answer that could be helpful. This would like follows:

def yield_from_pickle(pfin):
    with open(pfin, 'rb') as fhin:
        while True:
            try:
                yield pickle.load(fhin)
            except EOFError:
                break

The next thing is, that I wish to use this data in a PyTorch (1.0.1) Dataloader. From what I found in other answers, I must feed it a Dataset which you can subset, but which must contain __len__ and __getitem__. It could like this:

class TextDataset(Dataset):
    def __init__(self, pfin):
        self.pfin = pfin

    def __len__(self):
        # memory-lenient way but exhaust generator?
        return sum(1 for _ in self.yield_from_pickle())

    def __getitem__(self, index):
        # ???
        pass

    def yield_from_pickle(self):
        with open(self.pfin, 'rb') as fhin:
            while True:
                try:
                    yield pickle.load(fhin)
                except EOFError:
                    break

But I am not at all sure if this is even possible. How can I implement __len__ and __getitem__ in a sensible way? I don't think what I am doing with __len__ is a good idea because that'll exhaust the generator, and I have no idea at all how to safely implement __getitem__ while retaining the generator.

Is there a better way? To summarize, I want to build a Dataset that can be fed to PyTorch's Dataloader (because of its multiprocessing abilities) but in a memory-efficient way where I don't have to read the whole file into memory.

Output:
See my other answer for your options.

In short, you need to either preprocess each sample into separate files, or use a data format that does not need to be loaded fully into memory for reading.


Instruction: PyTorch: accuracy of validation set greater than 100% during training

Input:
1 ) Problem

I observe an odd behaviour during training where my validation-accuracy is above 100% right from the start.

Epoch 0/3
----------
100%|██████████| 194/194 [00:50<00:00,  3.82it/s]
train Loss: 1.8653 Acc: 0.4796
100%|██████████| 194/194 [00:32<00:00,  5.99it/s]
val Loss: 1.7611 Acc: 1.2939

Epoch 1/3
----------
100%|██████████| 194/194 [00:42<00:00,  4.61it/s]
train Loss: 0.8704 Acc: 0.7467
100%|██████████| 194/194 [00:31<00:00,  6.11it/s]
val Loss: 1.0801 Acc: 1.4694

The output indicates that one epoch iterates over 194 batches, which does seem to be  correct for the training data (which has a length of 6186, batch_size is 32, hence 32*194 = 6208 and this is ≈6186) but does not match the size of the validation-data (length of 3447, batch_size = 32).

Hence I would expect my validation-loop to generate 108 (3447 / 32 ≈ 108) batches insted of 194.

I thought this behaviour is handled within my for loop at:

for dataset in tqdm(dataloaders[phase]):

But somehow I can't figure out what is wrong here. See point 3) below for my entire code.

2 ) Question

If my assumption above is correct i.e. that this error stems from the for-loop within in my code then I would like to know the following:

How do I need to adjust the for-loop during the validation phase to handle the number of batches that are being used for validation correctly?

3 ) Background:

Following two tutorials, one on how to do transfer-learning (https://discuss.pytorch.org/t/transfer-learning-using-vgg16/20653) and one on how to do data-loading (https://pytorch.org/tutorials/beginner/data_loading_tutorial.html) in pytorch, I am trying to customize the code such that I can perform transfer-learning on a new custom dataset which I want to provide via pandas dataframes.

As such, my training- and validation-data is provided via two dataframes (df_train & df_val) which both contain two columns, one for the path and one for the target. E.g. like this:

    url                                 target
0   C:/Users/aaron/Desktop/pics/4ebd... 9
1   C:/Users/aaron/Desktop/pics/7153... 3
2   C:/Users/aaron/Desktop/pics/3ee6... 3
3   C:/Users/aaron/Desktop/pics/4652... 16
4   C:/Users/aaron/Desktop/pics/28ce... 15
...

And their respective length:

print(len(df_train))
print(len(df_val))
>> 6186
>> 3447

My pipeline looks like this:

class CustomDataset(Dataset):
    def __init__(self, df, transform=None):

        self.dataframe = df_train
        self.transform = transform

    def __len__(self):
        return len(self.dataframe)

    def __getitem__(self, idx):
        img_name = self.dataframe.iloc[idx, 0]
        img = Image.open(img_name)
        img_normalized = self.transform(img)

        landmarks = self.dataframe.iloc[idx, 1]
        sample = {'data': img_normalized, 'label': int(landmarks)}

        return sample

train_dataset = CustomDataset(df_train,transform=transforms.Compose([
                                                transforms.Resize(224),
                                               transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]))

val_dataset = CustomDataset(df_val,transform=transforms.Compose([
                                                transforms.Resize(224),
                                               transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]))

train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=32,shuffle=True, num_workers=0)
val_loader = torch.utils.data.DataLoader(val_dataset,batch_size=32,shuffle=True, num_workers=0)

dataloaders = {'train': train_loader, 'val': val_loader}
dataset_sizes = {'train': len(df_train) ,'val': len(df_val)}

################### Training

from tqdm import tqdm

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for dataset in tqdm(dataloaders[phase]):

                inputs, labels = dataset["data"], dataset["label"]
                #print(inputs.type())
                inputs = inputs.to(device, dtype=torch.float)
                labels = labels.to(device,dtype=torch.long)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, len(le.classes_))

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=4)

Output:
Your problem appears to be here:

class CustomDataset(Dataset):
    def __init__(self, df, transform=None):
>>>>>        self.dataframe = df_train

This should be

             self.dataframe = df

In your case, you are inadvertently setting both the train and val CustomDataset to df_train ...


Instruction: Torch Dataset Looping too far

Input:
Why does this dataset try to iterate past the final element

from torch.utils.data.dataset import Dataset
class DumbDataset(Dataset):
    def __init__(self, dct):
        self.dct = dct
        self.mapping = dict(enumerate(dct))
    def __getitem__(self, index):
        return self.dct[self.mapping[index]]

    def __len__(self):
        print('called')
        return len(self.dct)

ds = DumbDataset({'a': 'aword', 'b': 'another_words'})

for k in ds: print(k)

This raises KeyError: 2, which I don't understand since length of the object is 2. Shouldn't the iterator get StopIteration once it is exhausted?

Output:
The reason why your code raises KeyError is that Dataset does not implement __iter__() and thus when used in a for-loop Python falls back to starting at index 0 and calling __getitem__ until IndexError is raised, as discussed here. You can modify DumbDataset to work like this by having it raise an IndexError when the index is out of bounds

def __getitem__(self, index):
    if index >= len(self): raise IndexError
    return self.dct[self.mapping[index]]

and then your loop

for k in ds:
    print(k)

will work as you expected. On the other hand, the typical template for torch datasets is that you can either loop through them with indexing

for i in range(len(ds)):
    k = ds[k]
    print(k)

or that you wrap them in a DataLoader which returns elements in batches

generator = DataLoader(ds)
for k in generator:
    print(k)


Instruction: How to combine/stack tensors and combine dimensions in PyTorch?

Input:
I need to combine 4 tensors, representing greyscale images, of size [1,84,84], into a stack of shape [4,84,84], representing four greyscale images with each image represented as a "channel" in tensor style CxWxH.

I am using PyTorch.

I've tried using torch.stack and torch.cat but if one of these is the solution, I am not having luck figuring out the correct prep/methodology to get my results.

Thank you for any help.

import torchvision.transforms as T

class ReplayBuffer:
    def __init__(self, buffersize, batchsize, framestack, device, nS):
        self.buffer = deque(maxlen=buffersize)
        self.phi = deque(maxlen=framestack)
        self.batchsize = batchsize
        self.device = device

        self._initialize_stack(nS)

    def get_stack(self):
        #t =  torch.cat(tuple(self.phi),dim=0)
        t =  torch.stack(tuple(self.phi),dim=0)
        return t

    def _initialize_stack(self, nS):
        while len(self.phi) < self.phi.maxlen:
            self.phi.append(torch.tensor([1,nS[1], nS[2]]))

a = ReplayBuffer(buffersize=50000, batchsize=64, framestack=4, device='cuda', nS=[1,84,84])
print(a.phi)
s = a.get_stack()
print(s, s.shape)

The above code returns:

print(a.phi)

deque([tensor([ 1, 84, 84]), tensor([ 1, 84, 84]), tensor([ 1, 84, 84]), tensor([ 1, 84, 84])], maxlen=4)

print(s, s.shape)

tensor([[ 1, 84, 84],
        [ 1, 84, 84],
        [ 1, 84, 84],
        [ 1, 84, 84]]) torch.Size([4, 3])

But what I would like is the return to simply be [4, 84, 84]. I suspect this is quite simple but it's escaping me.

Output:
It seems you have misunderstood what torch.tensor([1, 84, 84]) is doing. Let's take a look:

torch.tensor([1, 84, 84])
print(x, x.shape) #tensor([ 1, 84, 84]) torch.Size([3])

You can see from the example above, it gives you a tensor with only one dimension.

From your problem statement, you need a tensor of shape [1,84,84].
Here's how it look like:

from collections import deque
import torch
import torchvision.transforms as T

class ReplayBuffer:
    def __init__(self, buffersize, batchsize, framestack, device, nS):
        self.buffer = deque(maxlen=buffersize)
        self.phi = deque(maxlen=framestack)
        self.batchsize = batchsize
        self.device = device

        self._initialize_stack(nS)

    def get_stack(self):
        t =  torch.cat(tuple(self.phi),dim=0)
#         t =  torch.stack(tuple(self.phi),dim=0)
        return t

    def _initialize_stack(self, nS):
        while len(self.phi) < self.phi.maxlen:
#             self.phi.append(torch.tensor([1,nS[1], nS[2]]))
            self.phi.append(torch.zeros([1,nS[1], nS[2]]))

a = ReplayBuffer(buffersize=50000, batchsize=64, framestack=4, device='cuda', nS=[1,84,84])
print(a.phi)
s = a.get_stack()
print(s, s.shape)

Note that torch.cat gives you a tensor of shape [4, 84, 84] and torch.stack gives you a tensor of shape [4, 1, 84, 84]. Their difference can be found at What's the difference between torch.stack() and torch.cat() functions?


Instruction: Training (DC)GAN, D(G(z)) goes to 0.5 while D(x) stays 0.9 and G(z) becomes corrupt

Input:
I'm currently training a DCGAN for 1x32x32 (channel, height, width) images.
Quite soon in training G(z) becomes reasonably realistic apart from a problem with the 'chessboard' artifacts being visible, but this should go away after lots of training?
However, after a long training session D(G(z)) goes to 0.5000 (and no longer changes) while D(x) stays between 0.8 and 0.9. Whenever D(G(z)) goes to 0.5 it also starts outputting fully black & white images. Hence, the generator no longer produces anything that looks close to what's in the training dataset. G(z) just becomes a black or white square.

The network used is from the original DCGAN paper, adapter for 1x32x32 images. With relu already replaced to leaky relu.

Output:
Solved the problem by switching to WGAN-GP (https://arxiv.org/abs/1704.00028).
Turns out it is more stable while training.


Instruction: How to set gradients to Zero without optimizer?

Input:
Between mutliple .backward() passes I'd like to set the gradients to zero. Right now I have to do this for every component seperately  (here these are x and t), is there a way to do this "globally" for all affected variables? (I imagine something like z.set_all_gradients_to_zero().)

I know there is optimizer.zero_grad() if you use an optimizer, but is there also a direct way without using an optimizer?

import torch

x = torch.randn(3, requires_grad = True)
t = torch.randn(3, requires_grad = True)
y = x + t
z = y + y.flip(0)

z.backward(torch.tensor([1., 0., 0.]), retain_graph = True)
print(x.grad)
print(t.grad)
x.grad.data.zero_()  # both gradients need to be set to zero
t.grad.data.zero_()
z.backward(torch.tensor([0., 1., 0.]), retain_graph = True)
print(x.grad)
print(t.grad)

Output:
You can also use nn.Module.zero_grad(). In fact, optim.zero_grad() just calls nn.Module.zero_grad() on all parameters which were passed to it.
There is no reasonable way to do it globally. You can collect your variables in a list
grad_vars = [x, t]
for var in grad_vars:
    var.grad = None

or create some hacky function based on vars(). Perhaps it's also possible to inspect the computation graph and zero the gradient of all leaf nodes, but I am not familiar with the graph API. Long story short, you're expected to use the object-oriented interface of torch.nn instead of manually creating tensor variables.


Instruction: pytorch torch.jit.trace returns function instead of torch.jit.ScriptModule

Input:
I need to run in c++ a pre-trained pytorch nn model (trained in python) to make predictions.

To do so, I'm following the instructions on how to load a pytorch model in c++ given here: https://pytorch.org/tutorials/advanced/cpp_export.html

But when I try to get the torch.jit.ScriptModule via tracing as stated in the first step of the tutorial:

    traced_script_module =
        torch.jit.trace(model, (input_tensor_1, input_tensor_2))

Instead of returning a torch.jit.ScriptModule, it returns a function:

    print(type(traced_script_module))
    <type 'function'>

Which, when I run:

    traced_script_module.save("model.pt")

then leads into the following error:

Traceback (most recent call last):
  File "serialize_model.py", line 60, in <module>
    traced_script_module.save("model.pt")
AttributeError: 'function' object has no attribute 'save'

Any ideas on what I'm doing wrong?

Output: Thanks for asking Jatentaki. I was using PyTorch 0.4 in Python and when I updated to 1.0 it worked.


Instruction: Expected input to torch Embedding layer with pre_trained vectors from gensim

Input:
I would like to use pre-trained embeddings in my neural network architecture. The pre-trained embeddings are trained by gensim. I found this informative answer which indicates that we can load pre_trained models like so:

import gensim
from torch import nn

model = gensim.models.KeyedVectors.load_word2vec_format('path/to/file')
weights = torch.FloatTensor(model.vectors)
emb = nn.Embedding.from_pretrained(torch.FloatTensor(weights.vectors))

This seems to work correctly, also on 1.0.1. My question is, that I don't quite understand what I have to feed into such a layer to utilise it. Can I just feed the tokens (segmented sentence)? Do I need a mapping, for instance token-to-index?

I found that you can access a token's vector simply by something like

print(weights['the'])
# [-1.1206588e+00  1.1578362e+00  2.8765252e-01 -1.1759659e+00 ... ]

What does that mean for an RNN architecture? Can we simply load in the tokens of the batch sequences? For instance:

for seq_batch, y in batch_loader():
    # seq_batch is a batch of sequences (tokenized sentences)
    # e.g. [['i', 'like', 'cookies'],['it', 'is', 'raining'],['who', 'are', 'you']]
    output, hidden = model(seq_batch, hidden)

This does not seem to work so I am assuming you need to convert the tokens to its index in the final word2vec model. Is that true? I found that you can get the indices of words by using the word2vec model's vocab:

weights.vocab['world'].index
# 147

So as an input to an Embedding layer, should I provide a tensor of int for a sequence of sentences that consist of a sequence of words? Example use with dummy dataloader (cf. example above) and dummy RNN welcome.

Output:
The documentation says the following

  This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.

So if you want to feed in a sentence, you give a LongTensor of indices, each corresponding to a word in the vocabulary, which the nn.Embedding layer will map into word vectors going forward.

Here's an illustration

test_voc = ["ok", "great", "test"]
# The word vectors for "ok", "great" and "test"
# are at indices, 0, 1 and 2, respectively.

my_embedding = torch.rand(3, 50)
e = nn.Embedding.from_pretrained(my_embedding)

# LongTensor of indicies corresponds to a sentence,
# reshaped to (1, 3) because batch size is 1
my_sentence = torch.tensor([0, 2, 1]).view(1, -1)

res = e(my_sentence)
print(res.shape)
# => torch.Size([1, 3, 50])
# 1 is the batch dimension, and there's three vectors of length 50 each

In terms of RNNs, next you can feed that tensor into your RNN module, e.g

lstm = nn.LSTM(input_size=50, hidden_size=5, batch_first=True)
output, h = lstm(res)
print(output.shape)
# => torch.Size([1, 3, 5])

I also recommend you look into torchtext. It can automatate some of the stuff you will have to do manually otherwise.


Instruction: how to load images data into pytorch dataLoader?

Input:
i am new to deep learning I want to use an algorithm written by pytorch, the example in pytorch tutorial is very specific . i have dataset in my Pc and i want to preprocess them .
thanks

Output:
class Get_Dataset( Dataset ):
    def __init__ (self) : #intital function define all member and  class variables
        super(Get_Dataset , self).__init__()
        scale = 255
        path = '/home/singhv/data/train/'
        trainA = os.listdir(path + 'image')
        trainB = os.listdir(path + 'mask')
        self.len = min([len(image) , len(mask)])
        self.object = np.ones((self.len ,128,128 , 3 ))
        self.target = np.ones((self.len , 128 , 128 , 3))
        print("Loading Dataset...")
        for i in tqdm(range(self.len)) :
            self.object[i] = cv2.resize(cv2.imread(path + 'image/' + image[i]),  (128,128))
            self.target[i] = cv2.resize(cv2.imread(path + 'mask/' + mask[i]),  (128,128))
        self.object =  torch.from_numpy(((self.object/(scale/2))-1 ) ).transpose_(3 , 1).double()
        self.target = torch.from_numpy(((self.target/(scale / 2)) -1 )).transpose_(3 , 1).double()

    def __getitem__(self , index   ) : # function to return dataset size
        return self.object[index] ,  self.target[index]

    def __len__(self): #you must have this function
        return self.len

train_dataset = Get_Dataset()
train_loader = DataLoader(train_dataset , batch_size = 1 , shuffle = True )


Instruction: Under macOS using pip install pytorch failure

Input:
When I'm using pip to install pytorch, some exception appeared.

Env:

Sys: MaxOS High Sierra
python version : 3.6
pip version : 19.0.2

input: pip install pytorch

output:

Collecting pytorch
  Using cached https://files.pythonhosted.org/packages/a9/41/4487bc23e3ac4d674943176f5aa309427b011e00607eb98899e9d951f67b/pytorch-0.1.2.tar.gz
Building wheels for collected packages: pytorch
  Building wheel for pytorch (setup.py) ... error
  Complete output from command /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-wheel-p0npaj5r --python-tag cp36:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py", line 17, in <module>
      raise Exception(message)
  Exception: You should install pytorch from http://pytorch.org

  ----------------------------------------
  Failed building wheel for pytorch
  Running setup.py clean for pytorch
Failed to build pytorch
Installing collected packages: pytorch
  Running setup.py install for pytorch ... error
    Complete output from command /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-record-us45ly0z/install-record.txt --single-version-externally-managed --compile:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py", line 13, in <module>
        raise Exception(message)
    Exception: You should install pytorch from http://pytorch.org

    ----------------------------------------
Command "/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-record-us45ly0z/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/

Output:
You're installing an old package named pytorch on PyPI i.e. pytorch 0.1.2. That's why you're receiving the exception.

You're supposed to install it from the pytorch website. There you'll an option to select your system configuration, it'll give you the command to install it. Also, the latest version of pytorch is named torch on PyPI. So, just do

pip3 install torch  # or with pip

If it fails due to cache, try with -no-cache-dir option.


Instruction: Confused about torch.nn.Sequential

Input:
Supposing we want to add a new layer, say a linear layer, to the end of the classifier of another model, such as VGG16, why exactly do these two implementations lead to different results? More specifically, I don't understand why the first implementation produces 2 classfiers:

vgg = torchvision.models.vgg16(pretrained=True)
vgg.classifer=nn.Sequential(vgg.classifier, nn.Linear(4096,300))
print(vgg)

output:

    VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
  (classifer): Sequential(
    (0): Sequential(
      (0): Linear(in_features=25088, out_features=4096, bias=True)
      (1): ReLU(inplace)
      (2): Dropout(p=0.5)
      (3): Linear(in_features=4096, out_features=4096, bias=True)
      (4): ReLU(inplace)
      (5): Dropout(p=0.5)
      (6): Linear(in_features=4096, out_features=1000, bias=True)
    )
    (1): Linear(in_features=4096, out_features=300, bias=True)
  )
)

implementation2:

vgg = models.vgg16(pretrained=True)
vgg=nn.Sequential(vgg, nn.Linear(4096,300))
print(vgg)

output:

Sequential(
  (0): VGG(
    (features): Sequential(
      (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (1): ReLU(inplace)
      (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (3): ReLU(inplace)
      (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (6): ReLU(inplace)
      (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (8): ReLU(inplace)
      (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (11): ReLU(inplace)
      (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (13): ReLU(inplace)
      (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (15): ReLU(inplace)
      (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (18): ReLU(inplace)
      (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (20): ReLU(inplace)
      (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (22): ReLU(inplace)
      (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (25): ReLU(inplace)
      (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (27): ReLU(inplace)
      (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (29): ReLU(inplace)
      (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    )
    (classifier): Sequential(
      (0): Linear(in_features=25088, out_features=4096, bias=True)
      (1): ReLU(inplace)
      (2): Dropout(p=0.5)
      (3): Linear(in_features=4096, out_features=4096, bias=True)
      (4): ReLU(inplace)
      (5): Dropout(p=0.5)
      (6): Linear(in_features=4096, out_features=1000, bias=True)
    )
  )
  (1): Linear(in_features=4096, out_features=300, bias=True)
)

Output:
It's because you have a syntax error in the spelling of classifier. You have written it as

vgg.classifer=nn.Sequential(vgg.classifier, nn.Linear(4096,300))

Note the missing i after f in classifier on LHS. So, you're inadvertently creating a new group of layers named classifer by this line.

After correction:

vgg.classifier=nn.Sequential(vgg.classifier, nn.Linear(4096,300))

Moreover, in the first example, you're replacing the existing classifier layer with a sequential network having classifier portion of original vgg and Linear layer as the last layer.

In the second example, you're recreating the variable vgg with a new sequential network which contains the original vgg network with the addition of Linear layer as the last layer.

vgg=nn.Sequential(vgg, nn.Linear(4096,300))

Note the difference between the above two.


Instruction: Pytorch: Why loss functions are implemented both in nn.modules.loss and nn.functional module?

Input:
Many loss functions in Pytorch are implemented both in nn.modules.loss and nn.functional.

For example, the two lines of the below return same results.

import torch.nn as nn
import torch.functional as F
nn.L1Loss()(x,y)
F.l1_loss(x,y)

Why are there two implementations?

Consistency for other parametric loss functions
Instantiation of loss function brings something good
otherwise

Output:
I think of it as of a partial application situation - it's useful to be able to "bundle" many of the configuration variables with the loss function object. In most cases, your loss function has to take prediction and ground_truth as its arguments. This makes for a fairly uniform basic API of loss functions. However, they differ in details. For instance, not every loss function has a reduction parameter. BCEWithLogitsLoss has weight and pos_weight parameters; PoissonNLLLoss has log_input, eps. It's handy to write a function like

def one_epoch(model, dataset, loss_fn, optimizer):
    for x, y in dataset:
        model.zero_grad()
        y_pred = model(x)
        loss = loss_fn(y_pred, y)
        loss.backward()
        optimizer.step()

which can work with instantiated BCEWithLogitsLoss equally well as with PoissonNLLLoss. But it cannot work with their functional counterparts, because of the bookkeeping necessary. You would instead have to first create

loss_fn_packed = functools.partial(F.binary_cross_entropy_with_logits, weight=my_weight, reduction='sum')

and only then you can use it with one_epoch defined above. But this packing is already provided with the object-oriented loss API, along with some bells and whistles (since losses subclass nn.Module, you can use forward and backward hooks, move stuff between cpu and gpu, etc).


Instruction: How to give a batch of frames to the model in pytorch c++ api?

Input:
I've written a code to load the pytorch model in C++ with help of the PyTorch C++ Frontend api. I want to give a batch of frames to a pretrained model in the C++ by using module->forward(batch_frames). But it can forward through a single input.
How can I give a batch of inputs to the model?

A part of code that I want to give the batch is shown below:

 cv::Mat frame;
 vector<torch::jit::IValue> frame_batch;

 // do some pre-processes on each frame and then add it to the frame_batch

 //forward through the batch frames
 torch::Tensor output = module->forward(frame_batch).toTensor();

Output:
Finally, I used a function in c++ to concatenate images and make a batch of images. Then convert the batch into the torch::tensor and feed the model using the batch. A part of code is given below:
// cat 2 or more images to make a batch
cv::Mat batch_image;
cv::vconcat(image_2, images_1, batch_image);

// do some pre-process on image
auto input_tensor_batch = torch::from_blob(batch_image.data, {size_of_batch, image_height, image_width, 3});
input_tensor_batch = input_tensor_batch.permute({0, 3, 1, 2});

//forward through the batch frames
 torch::Tensor output = module->forward({input_tensor_batch}).toTensor();

Note that put the { } in the forward-pass function!


Instruction: pytorch: How does loss behave when coming from two networks?

Input:
I am trying to implement the following algorithm in this book, section 13.5, in pytorch.

This would require two separate neural networks, (in this question, model1 and model2). One's loss is dependent only on its own output [via delta] (parameterized by w), the other (parameterized by theta), dependent both on its own output [via ln(pi)], and on the other's output [again, via delta].

I want to update each one separately

Assume the following models implement nn.Module:

model1 = Mynet1()
model2 = Mynet2()

val1 = model1(input1)
val2 = model2(input2)

self.optimizer1 = optim.Adam(model1.parameters(), lr1)
self.optimizer2 = optim.Adam(model2.parameters(), lr2)

loss1 = f(val1)
loss2 = f(val1, val2)#THIS IS THE INTERESTING PART

optim1.zero_grad()
loss1.backward
optim1.step()

optim2.zero_grad()
loss2.backward
optim2.step()

I understand that applying backward on loss1, then stepping its optimizer would update model1's parameters.

My question is what happens when activating the same on loss2, model2, optimizer2, where loss 2 is dependant on outputs both from model1 and model2?

How can I make the loss2 update not affect model1 parameters?

Output:
since optim2 has only model2's parameter it will only update model2 if you do optim2.step() as is being done.

However, loss2.backward() will compute gradients for both model1 and model2's params and if you do optim1.step() after that it will update model1's params. If you don't want to compute gradients for model1's param, you can do val1.detach() to detach it from the computational graph.


Instruction: Pytorch ValueError: optimizer got an empty parameter list

Input:
When trying to create a neural network and optimize it using Pytorch, I am getting

ValueError: optimizer got an empty parameter list

Here is the code.
import torch.nn as nn
import torch.nn.functional as F
from os.path import dirname
from os import getcwd
from os.path import realpath
from sys import argv

class NetActor(nn.Module):
    def __init__(self, args, state_vector_size, action_vector_size, hidden_layer_size_list):
        super(NetActor, self).__init__()
        self.args = args

        self.state_vector_size = state_vector_size
        self.action_vector_size = action_vector_size
        self.layer_sizes = hidden_layer_size_list
        self.layer_sizes.append(action_vector_size)

        self.nn_layers = []
        self._create_net()

    def _create_net(self):
        prev_layer_size = self.state_vector_size
        for next_layer_size in self.layer_sizes:
            next_layer = nn.Linear(prev_layer_size, next_layer_size)
            prev_layer_size = next_layer_size
            self.nn_layers.append(next_layer)

    def forward(self, torch_state):
        activations = torch_state
        for i,layer in enumerate(self.nn_layers):
            if i != len(self.nn_layers)-1:
                activations = F.relu(layer(activations))
            else:
                activations = layer(activations)

        probs = F.softmax(activations, dim=-1)
        return probs

and then the call
        self.actor_nn = NetActor(self.args, 4, 2, [128])
        self.actor_optimizer = optim.Adam(self.actor_nn.parameters(), lr=args.learning_rate)

gives the very informative error

ValueError: optimizer got an empty parameter list

I find it hard to understand what exactly in the network's definition makes the network have parameters.
I am following and expanding the example I found in Pytorch's tutorial code.
I can't really tell the difference between my code and theirs that makes mine think it has no parameters to optimize.
How to make my network have parameters like the linked example?

Output:
Your NetActor does not directly store any nn.Parameter. Moreover, all other layers it eventually uses in forward are stored as a simple list in self.nn_layers.
If you want self.actor_nn.parameters() to know that the items stored in the list self.nn_layers may contain trainable parameters, you should work with containers.
Specifically, making self.nn_layers to be a nn.ModuleList instead of a simple list should solve your problem:
self.nn_layers = nn.ModuleList()


Instruction: How to find the max index for each row in a tensor object?

Input:
So I'm creating a pytorch model and for the forward pass, I'm applying my forward pass method to get the scores tensor which contains the prediction scores for each class. The shape of this tensor is [100, 10]. Now, I want to get the accuracy by comparing it to y which contains the actual scores. This tensor has the shape [100]. To compare the two I'll be using torch.mean(scores == y) and I'll count how many are the same.

The problem is that I need to convert the scores tensor so that each row simply contains the index of the highest value in each row. For example if the tensor looked like this,

tensor(
    [[0.3232, -0.2321, 0.2332, -0.1231, 0.2435, 0.6728],

    [0.2323, -0.1231, -0.5321, -0.1452, 0.5435, 0.1722],

    [0.9823, -0.1321, -0.6433, 0.1231, 0.023, 0.0711]]
)

Then I'd want it to be converted so that it looks like this.

tensor([5, 4, 0])

How could I do that?

Output:
Use argmax with desired dim (a.k.a. axis)

a = tensor(
    [[0.3232, -0.2321, 0.2332, -0.1231, 0.2435, 0.6728],
    [0.2323, -0.1231, -0.5321, -0.1452, 0.5435, 0.1722],
    [0.9823, -0.1321, -0.6433, 0.1231, 0.023, 0.0711]]
)

a.argmax(1)
# tensor([ 5,  4,  0])


Instruction: requires_grad of params is True even with torch.no_grad()

Input:
I am experiencing a strange problem with PyTorch today.

When checking network parameters in the with scope, I am expecting requires_grad to be False, but apparently this is not the case unless I explicitly set all params myself.

Code

Link to Net -> Gist

net = InceptionResnetV2()

with torch.no_grad():

    for name, param in net.named_parameters():
        print("{} {}".format(name, param.requires_grad))

The above code will tell me all the params are still requiring grad, unless I explicitly specify param.requires_grad = False.

My torch version: 1.0.1.post2

Output:
torch.no_grad() will disable gradient information for the results of operations involving tensors that have their requires_grad set to True. So consider the following:

import torch

net = torch.nn.Linear(4, 3)

input_t = torch.randn(4)

with torch.no_grad():

    for name, param in net.named_parameters():
        print("{} {}".format(name, param.requires_grad))

    out = net(input_t)

    print('Output: {}'.format(out))

    print('Output requires gradient: {}'.format(out.requires_grad))
    print('Gradient function: {}'.format(out.grad_fn))

This prints

weight True
bias True
Output: tensor([-0.3311,  1.8643,  0.2933])
Output requires gradient: False
Gradient function: None

If you remove with torch.no_grad(), you get

weight True
bias True
Output: tensor([ 0.5776, -0.5493, -0.9229], grad_fn=<AddBackward0>)
Output requires gradient: True
Gradient function: <AddBackward0 object at 0x7febe41e3240>

Note that in both cases the module parameters have requires_grad set to True, but in the first case the out tensor doesn't have a gradient function associated with it whereas it does in the second case.


Instruction: PyTorch VAE fails conversion to onnx

Input:
I'm trying to convert a PyTorch VAE to onnx, but I'm getting: torch.onnx.symbolic.normal does not exist

The problem appears to originate from a reparametrize() function:

    def reparametrize(self, mu, logvar):
        std = logvar.mul(0.5).exp_()
        if self.have_cuda:
             eps = torch.normal(torch.zeros(std.size()),torch.ones(std.size())).cuda()
        else:
           eps = torch.normal(torch.zeros(std.size()),torch.ones(std.size()))
        return eps.mul(std).add_(mu)

I also tried:

eps = torch.cuda.FloatTensor(std.size()).normal_()

which produced the error:

    Schema not found for node. File a bug report.
    Node: %173 : Float(1, 20) = aten::normal(%169, %170, %171, %172), scope: VAE
    Input types:Float(1, 20), float, float, Generator

and

eps = torch.randn(std.size()).cuda()

which produced the error:

    builtins.TypeError: i_(): incompatible function arguments. The following argument types are supported:
    1. (self: torch._C.Node, arg0: str, arg1: int) -> torch._C.Node
    Invoked with: %137 : Tensor = onnx::RandomNormal(), scope: VAE, 'shape', 133 defined in (%133 : int[] = prim::ListConstruct(%128, %132), scope: VAE) (occurred when translating randn)

I am using cuda.

Any thoughts appreciated. Perhaps I need to approach the z/latent differently for onnx?

NOTE: Stepping through, I can see that it's finding RandomNormal() for torch.randn(), which should be correct. But I don't really have access to the arguments at that point, so how can I fix it?

Output:
In very short, the code bellow may work. (at least in my environment, it worked w/o errors).

It seems that .size() operator might return variable, not constant, so it causes error for onnx compilation. (I got the same error when changed to use .size())

import torch
import torch.utils.data
from torch import nn
from torch.nn import functional as F

IN_DIMS = 28 * 28
BATCH_SIZE = 10
FEATURE_DIM = 20

class VAE(nn.Module):
    def __init__(self):
        super(VAE, self).__init__()

        self.fc1 = nn.Linear(784, 400)
        self.fc21 = nn.Linear(400, FEATURE_DIM)
        self.fc22 = nn.Linear(400, FEATURE_DIM)
        self.fc3 = nn.Linear(FEATURE_DIM, 400)
        self.fc4 = nn.Linear(400, 784)

    def encode(self, x):
        h1 = F.relu(self.fc1(x))
        return self.fc21(h1), self.fc22(h1)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5*logvar)
        eps = torch.randn(BATCH_SIZE, FEATURE_DIM, device='cuda')
        return eps.mul(std).add_(mu)

    def decode(self, z):
        h3 = F.relu(self.fc3(z))
        return torch.sigmoid(self.fc4(h3))

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        recon_x = self.decode(z)

        return recon_x

model = VAE().cuda()

dummy_input = torch.randn(BATCH_SIZE, IN_DIMS, device='cuda')
torch.onnx.export(model, dummy_input, "vae.onnx", verbose=True)


Instruction: Moving member tensors with module.to() in PyTorch

Input:
I am building a Variational Autoencoder (VAE) in PyTorch and have a problem writing device agnostic code. The Autoencoder is a child of nn.Module with an encoder and decoder network, which are too. All weights of the network can be moved from one device to another by calling net.to(device).

The problem I have is with the reparametrization trick:

encoding = mu + noise * sigma

The noise is a tensor of the same size as mu and sigma and saved as a member variable of the autoencoder module. It is initialized in the constructor and resampled in-place each training step. I do it that way to avoid constructing a new noise tensor each step and pushing it to the desired device. Additionally, I want to fix the noise in the evaluation. Here is the code:

class VariationalGenerator(nn.Module):
    def __init__(self, input_nc, output_nc):
        super(VariationalGenerator, self).__init__()

        self.input_nc = input_nc
        self.output_nc = output_nc
        embedding_size = 128

        self._train_noise = torch.randn(batch_size, embedding_size)
        self._eval_noise = torch.randn(1, embedding_size)
        self.noise = self._train_noise

        # Create encoder
        self.encoder = Encoder(input_nc, embedding_size)
        # Create decoder
        self.decoder = Decoder(output_nc, embedding_size)

    def train(self, mode=True):
        super(VariationalGenerator, self).train(mode)
        self.noise = self._train_noise

    def eval(self):
        super(VariationalGenerator, self).eval()
        self.noise = self._eval_noise

    def forward(self, inputs):
        # Calculate parameters of embedding space
        mu, log_sigma = self.encoder.forward(inputs)
        # Resample noise if training
        if self.training:
            self.noise.normal_()
        # Reparametrize noise to embedding space
        inputs = mu + self.noise * torch.exp(0.5 * log_sigma)
        # Decode to image
        inputs = self.decoder(inputs)

        return inputs, mu, log_sigma

When I now move the autoencoder to the GPU with net.to('cuda:0') I get an error in forwarding because the noise tensor is not moved.

I don't want to add a device parameter to the constructor, because then it is still not possible to move it to another device later. I also tried to wrap the noise into nn.Parameter so that it is affected by net.to(), but that gives an error from the optimizer, as the noise is flagged as requires_grad=False.

Anyone has a solution to move all of the modules with net.to()?

Output:
A better version of tilman151's second approach is probably to override _apply, rather than to. That way net.cuda(), net.float(), etc will all work as well, since those all call _apply rather than to (as can be seen in the source, which is simpler than you might think):

def _apply(self, fn):
    super(VariationalGenerator, self)._apply(fn)
    self._train_noise = fn(self._train_noise)
    self._eval_noise = fn(self._eval_noise)
    return self


Instruction: Add channel to MNIST via transform?

Input:
I'm trying to use the MNIST dataset from torchvision.datasets.It seems to be provided as an N x H x W (uint8) (batch dimension, height, width) tensor. All the pytorch classes for working on images (for instance Conv2d) however require a N x C x H x W (float32) tensor where C is the number of colour channels. I've tried to add add the ToTensor transform but that didn't add a color channel.

Is there a way using torchvision.transforms to add this additional dimension? For a raw tensor we could just do .unsqueeze(1) but that doesn't look like a very elegant solution. I'm just trying to do it the "proper" way.

Here is the failed conversion.

import torchvision
dataset = torchvision.datasets.MNIST("~/PyTorchDatasets/MNIST/", train=True, transform=torchvision.transforms.ToTensor(), download=True)
print(dataset.train_data[0])

Output:
I had a misconception: dataset.train_data is not affected by the specified transform, only the output of a DataLoader(dataset,...) will be. After checking data from

for data, _ in DataLoader(dataset):
    break

we can see that ToTensor actually does exactly what is desired.


Instruction: How to do gradient clipping in pytorch?

Input:
What is the correct way to perform gradient clipping in pytorch?
I have an exploding gradients problem.

Output:
clip_grad_norm (which is actually deprecated in favor of clip_grad_norm_ following the more consistent syntax of a trailing _ when in-place modification is performed) clips the norm of the overall gradient by concatenating all parameters passed to the function, as can be seen from the documentation:

The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place.

From your example it looks like that you want clip_grad_value_ instead which has a similar syntax and also modifies the gradients in-place:
clip_grad_value_(model.parameters(), clip_value)

Another option is to register a backward hook. This takes the current gradient as an input and may return a tensor which will be used in-place of the previous gradient, i.e. modifying it. This hook is called each time after a gradient has been computed, i.e. there's no need for manually clipping once the hook has been registered:
for p in model.parameters():
    p.register_hook(lambda grad: torch.clamp(grad, -clip_value, clip_value))


Instruction: PyTorch: What does @weak_script_method decorator do?

Input:
In the torch.nn.Linear class (and other classes too), the forward method includes a @weak_script_method decorator as follows:

@weak_script_method
def forward(self, input):
    return F.linear(input, self.weight, self.bias)

What does this decorator do? Should I include it if I'm overriding the forward method in my own subclass of the Linear module?

Output:
You can find the exact decorator location to get the idea.

def weak_script_method(fn):
    weak_script_methods[fn] = {
        "rcb": createResolutionCallback(frames_up=2),
        "original_method": fn
    }
return fn

But, you shouldn't need to worry about that decorator. This decorator is internal to JIT.

Technically method decorated with @weak_script_method will be added to the weak_script_methods dictionary created in front, like this:

weak_script_methods = weakref.WeakKeyDictionary()

That dict tracks methods to avoid circular dependency problems; methods calling other methods while creating the PyTorch graph.

This really has no much sense unless you understand the concept of TorchScript in general.

The idea of TorchScript is to train models in PyTorch and export models to another non Python production environment (read:C++/C/Cuda) that support static typing.

PyTorch team made TorchScript on limited Python base to support static typing.
By default, Python is dynamically typed language, but with few tricks (read:checks) it can become statically typed language.

And so TorchScript functions are statically-typed subset of Python that contains all of PyTorch's built-in Tensor operations. This difference allows TorchScript modules code to run without the need for a Python interpreter.

You can either convert the existing PyTorch methods to TorchScript using tracing (torch.jit.trace() method), or to create your TorchScripts by hand using @torch.jit.script decorator.

If you use tracing you will get a single class module at the end. Here is the example:

import inspect

import torch
def foo(x, y):
    return x + y
traced_foo = torch.jit.trace(foo, (torch.rand(3), torch.rand(3)))

print(type(traced_foo)) #<class 'torch.jit.TopLevelTracedModule'>
print(traced_foo) #foo()
print(traced_foo.forward) #<bound method TopLevelTracedModule.forward of foo()>

lines = inspect.getsource(traced_foo.forward)
print(lines)

Output:

<class 'torch.jit.TopLevelTracedModule'>
foo()
<bound method TopLevelTracedModule.forward of foo()>
    def forward(self, *args, **kwargs):
        return self._get_method('forward')(*args, **kwargs)

You can investigate further using the inspect module. This was just a showcase how to convert one function using tracing.