Instruction: PyTorch non-deterministic dropout

Input:
I'm trying to make output of BLSTM deterministic, after investigation its appeared that my dropout layer creates not deterministic dropout masks, so I was researching about how to fix random seed in pytorch. I found this page and other suggestions though I put everything in code it did not help. Here is my code:

import sys
import random
import datetime as dt

import numpy as np
import torch

torch.manual_seed(42)
torch.cuda.manual_seed(42)
np.random.seed(42)
random.seed(42)
torch.backends.cudnn.deterministic = True

ex = torch.ones(10)
torch.nn.functional.dropout(ex, p=0.5, training=True)
# Out[29]: tensor([0., 0., 2., 0., 0., 0., 0., 0., 2., 2.])

torch.nn.functional.dropout(ex, p=0.5, training=True)
# Out[30]: tensor([0., 2., 0., 2., 2., 0., 0., 2., 2., 2.])

Please help me get deterministic output from dropout for the same input

Output:
every time you want the same output you need to reset the random seed again so:

>>> import torch

>>> torch.manual_seed(42)
<torch._C.Generator object at 0x127cd9170>

>>> ex = torch.ones(10)

>>> torch.nn.functional.dropout(ex, p=0.5, training=True)
tensor([0., 0., 2., 2., 2., 2., 2., 0., 2., 0.])

>>> torch.manual_seed(42)
<torch._C.Generator object at 0x127cd9170>

>>> torch.nn.functional.dropout(ex, p=0.5, training=True)
tensor([0., 0., 2., 2., 2., 2., 2., 0., 2., 0.])

You may want to keep resetting all those random seeds in general though, you can find a lot of different randomness when building neural nets in python


Instruction: How to efficiently calculate pairwise intersection of nonzero indices in a scipy.csr sparse matrix?

Input:
I have a scipy.sparse.csr matrix X which is n x p. For each row in X I would like to compute the intersection of the non zero element indices with each row in X and store them in a new tensor or maybe even a dictionary. For example, X is:

X = [
[0., 1.5, 4.7],
[4., 0., 0.],
[0., 0., 2.6]
]

I would like the output to be

intersect =
[
[[1,2], [], [2]],
[[], [0], []],
[[2], [], [2]]
]

intersect[i,j] is an ndarray representing the intersection of the indices of nonzero elements of ith and jth rows of X i.e X[i], X[j].

Currently the way I am doing this is by looping and I would like to vectorize this as it would be much faster and the computations are done in parallel.

# current code
n = X.shape[0]
intersection_dict = {}
for i in range(n):
    for j in range(n):
        indices = np.intersect1d(X[i].indices, X[j].indices)
        intersection_dict[(i,j)] = indices

My n is pretty large so looping over n^2 is very poor. I am just having trouble figuring out a way to vectorize this operation. Does anybody have any ideas on how to tackle this?

EDIT:
It was made apparent that I should explain the problem I am trying to solve, so here it is.

I am solving an optimization problem and have an equation
W = X diag(theta) X'. I want to find W in a quick manner as I update the entries of theta till convergence. Further I am updating parameters using pytorch where sparse operations are not as extensive as in scipy.

where:

X : n x p sparse data matrix (n documents, p features)
theta : p x 1 parameter vector I want to learn and will be updating
X' : p x n transpose of sparse data matrix

note p >> n

I had in mind two methods of solving this quickly

Cache sparse outer product of (see More efficient matrix multiplication with diagonal matrix)
W_ij = X_i * theta * X_j (element wise product of row i of X, theta, and row j of X. And since X_i, X_j are sparse I was thinking if I take the intersection of the nonzero indices then I can do a simple dense elementwise product (sparse element wise product not supported in pytorch) of X_i[intersection indices] * theta[intersection indices] X_j[intersection indices]

I want to vectorize as much of this computation as possible rather than loop as my n is typically in the thousands and p is 11 million.

I am attempting method 2 over method 1 do to the lack of sparse support in Pytorch. Mainly when updating the entries of theta I would not like to do sparse-dense or sparse-sparse operations. I want to do dense-dense operations.

Output:
The optimization you're looking at requires storing p different n x n matrices. If you do want to try it, I'd probably use all the functionality built into sparse matrices in scipy's C extensions.

import numpy as np
from scipy import sparse

arr = sparse.random(100,10000, format="csr", density=0.01)

xxt = arr @ arr.T
p_comps = [arr[:, i] @ arr.T[i, :] for i in range(arr.shape[1])]

def calc_weights(xxt, thetas, p_comps):
    xxt = xxt.copy()
    xxt.data = np.zeros(xxt.data.shape, dtype=xxt.dtype)
    for i, t in enumerate(thetas):
        xxt += (p_comps[i] * t)
    return xxt

W = calc_weights(xxt, np.ones(10000), p_comps)

>>>(xxt.A == W.A).all()
True

It's really unlikely that this is going to work well implemented in python. You may have better luck doing this in C, or writing something with nested loops that operates on elements and is amenable to getting JIT compiled with numba.


Instruction: How to solve size mismatch error in pytorch?

Input:
I am trying to create a logistic model by using CIFAR10 data in PyTorch. After running the model for evaluation I run into an error :

  RuntimeError: size mismatch, m1: [750 x 4096], m2: [1024 x 10] at C:\w\1\s\tmp_conda_3.7_100118\conda\conda-bld\pytorch_1579082551706\work\aten\src\TH/generic/THTensorMath.cpp:136

It seems like input_size is creating a problem, I dont know I am new to this. Please let me know what changes should I make in order to overcome this error.

These are the hyperparameters:

batch_size = 100
learning_rate = 0.001

# Other constants
input_size = 4*4*64
num_classes = 10

This is the cell that downloads and splits the dataset into train, validation and test.

transform = torchvision.transforms.Compose(
    [torchvision.transforms.ToTensor(),
     torchvision.transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))])

testset = torchvision.datasets.CIFAR10(root='D:\PyTorch\cifar-10-python', train=False,download=False, transform=transform)
trainvalset = torchvision.datasets.CIFAR10(root='D:\PyTorch\cifar-10-python', train=True,download=False, transform=transform)
trainset, valset = torch.utils.data.random_split(trainvalset, [45000, 5000]) # 10% for validation

train_loader = torch.utils.data.DataLoader(trainset, batch_size=50, shuffle=True)
test_loader = torch.utils.data.DataLoader(testset, batch_size=1000, shuffle=False)
val_loader = torch.utils.data.DataLoader(valset, batch_size=1000, shuffle=False)

This is the architecture of my model.

class CifarModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size,  num_classes)
    def forward(self, xb):
        xb = xb.view(-1, 64*8*8)
        #xb = xb.reshape(-1, 784)
        print(xb.shape)
        out = self.linear(xb)
        return out

    def training_step(self, batch):
        images, labels = batch
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        return loss

    def validation_step(self, batch):
        images, labels = batch
        out = self(images)                    # Generate predictions
        loss = F.cross_entropy(out, labels)   # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss.detach(), 'val_acc': acc.detach()}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))

model = CifarModel()

def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))
def evaluate(model, val_loader):
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result)
        history.append(result)
    return history
evaluate(model, val_loader)

Output:
Here you are specifying that the number of output classes should be 10:

num_classes = 10

Your forward function does not reflect this:

xb = xb.view(-1, 64*8*8) # you get 750x4096
out = self.linear(xb) # here an input of
# input_size to linear layer = 4*4*64 # 1024
# num_classes = 10

Modify it like this:

xb = xb.view(-1, 64*4*4) # you get 750x1024
out = self.linear(xb) # M1 750x1024 M2 1024x10:
# input_size = 4*4*64 # 1024
# num_classes = 10


Instruction: How to solve error: no match between expected input batch size and target batch size in PyTorch?

Input:
I attempting to create a logistic model on CIFAR10 dataset by PyTorch. However I am getting an error:

  ValueError: Expected input batch_size (900) to match target batch_size (300).

What I think is happening is that 3*100 is 300. So may be the 3 axis of the RGB image is doing that but I cant figure how to solve.

These are my hyperparameters.

batch_size = 100
learning_rate = 0.001

# Other constants
input_size = 32*32
num_classes = 10

Here I divide my data into train, validation and test data.

transform_train = transforms.Compose([transforms.Resize((32,32)),
                                      transforms.RandomHorizontalFlip(),
                                      transforms.RandomRotation(10),
                                      transforms.RandomAffine(0, shear=10, scale=(0.8,1.2)),
                                      transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),
                                      transforms.ToTensor(),
                                      transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ])

transform = transforms.Compose([transforms.Resize((32,32)),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ])
training_dataset = CIFAR10(root='D:\PyTorch\cifar-10-python', train=True, download=True, transform=transform_train)
train_ds, val_ds = random_split(training_dataset, [40000, 10000])
test_ds = CIFAR10(root='D:\PyTorch\cifar-10-python', train=False, download=True, transform=transform)

train_loader = DataLoader(train_ds, batch_size=100, shuffle=True)
val_loader = DataLoader(val_ds, batch_size = 100, shuffle = False)
test_loader = DataLoader(test_ds, batch_size = 100, shuffle=False)

This is the model.

class CifarModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size,  num_classes)
    def forward(self, xb):
        xb = xb.view(-1, 32*32)
        #xb = xb.reshape(-1, 784)
        print(xb.shape)
        out = self.linear(xb)
        return out

    def training_step(self, batch):
        images, labels = batch
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        return loss

    def validation_step(self, batch):
        images, labels = batch
        out = self(images)                    # Generate predictions
        loss = F.cross_entropy(out, labels)   # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss.detach(), 'val_acc': acc.detach()}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))

model = CifarModel()

def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))
def evaluate(model, val_loader):
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result)
        history.append(result)
    return history
evaluate(model, val_loader)

Here's the error I encounter when I run evaluate function:

torch.Size([900, 1024])

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-23-3621eab8de1a> in <module>
     21         history.append(result)
     22     return history
---> 23 evaluate(model, val_loader)

<ipython-input-23-3621eab8de1a> in evaluate(model, val_loader)
      3     return torch.tensor(torch.sum(preds == labels).item() / len(preds))
      4 def evaluate(model, val_loader):
----> 5     outputs = [model.validation_step(batch) for batch in val_loader]
      6     return model.validation_epoch_end(outputs)
      7

<ipython-input-23-3621eab8de1a> in <listcomp>(.0)
      3     return torch.tensor(torch.sum(preds == labels).item() / len(preds))
      4 def evaluate(model, val_loader):
----> 5     outputs = [model.validation_step(batch) for batch in val_loader]
      6     return model.validation_epoch_end(outputs)
      7

<ipython-input-22-c9e17d21eaff> in validation_step(self, batch)
     19         images, labels = batch
     20         out = self(images)                    # Generate predictions
---> 21         loss = F.cross_entropy(out, labels)   # Calculate loss
     22         acc = accuracy(out, labels)           # Calculate accuracy
     23         return {'val_loss': loss.detach(), 'val_acc': acc.detach()}

~\Anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

~\Anaconda3\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1834     if input.size(0) != target.size(0):
   1835         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
   1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (900) to match target batch_size (300).

Output:
One problem that I am seeing is this line:

xb = xb.view(-1, 32*32)

Here you are saying that the input image has only one channels. In other words, grayscale. Change it to reflect the number of channels (RGB):

xb = xb.view(-1, 32*32*3)


Instruction: How to extract output of torch model in c++?

Input:
I have got trained keras model and converted it using mmdnn. Then I try use it in c++ code:

#include <iostream>

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

#include <torch.h>

int main()
{
    cv::Mat image;
    image= cv::imread("test_img.png", cv::IMREAD_GRAYSCALE);   // Read the file

try
{
    torch::jit::script::Module module;
    module = torch::jit::load("my_model.pth");

    torch::IntArrayRef input_dim = std::vector<int64_t>({ 1, 2, 256, 256});

    cv::Mat input_img;
    image.convertTo(input_img, CV_32FC3, 1 / 255.0);
    torch::Tensor x = torch::from_blob(input_img.data, { 1, 2, 256, 256 }, torch::kFloat);
    torch::NoGradGuard no_grad;

    auto output = module.forward({ x });

    float* data = static_cast<float*>(output.toTensor().data_ptr());

    cv::Mat output_img = cv::Mat(256, 256, CV_32FC3, data);
    cv::imwrite("output_img.png", output_img);
}
catch (std::exception &ex)
{
    std::cout << "exception! " << ex.what() << std::endl;
}

    return 0;
}

This code throws an exception:

  exception! isTensor() INTERNAL ASSERT FAILED at
  E:\20B\pytorch\pytorch\aten\src\ATen/core/ivalue_inl.h:112, please
  report a bug to PyTorch. Expected Tensor but got Tuple (toTensor at
  E:\20B\pytorch\pytorch\aten\src\ATen/core/ivalue_inl.h:112) (no
  backtrace available)

This was thrown in line float* data = static_cast<float*>(output.toTensor().data_ptr()); when the function toTensor() was called. If I use toTuple() instead of toTensor() then the result doesn't have the function data_ptr(), but I need this for extracting data (and putting it into opencv image).

How to extract image from the model output?

Output:
In this case the answer of model is tuple of 2 images. We can extract them by such way:

torch::Tensor t0 = output.toTuple()->elements()[0].toTensor();
torch::Tensor t1 = output.toTuple()->elements()[1].toTensor();

Variables t0 and t1 contain tensors with output of model.


Instruction: Pytorch: Recover network with customized VGG model that was saved improperly

Input:
I am currently doing work with customizing the forward method for models. I was using some tutorial code that ran VGG. I did a few runs with the baseline model and it seemed to work fine. Afterwards, I replaced the forward method for the VGG using:

net.forward = types.MethodType(forward_vgg_new, net)

Unfortunately, the way that the tutorial code saves the models is:

            state = {
                    'net':net,
                    'acc':acc,
                    'epoch':epoch,
            }
...
            torch.save(state, ...)

While This worked for the original tutorial code, loading no longer works for my custom models as I get:

  AttributeError: 'VGG' object has no attribute 'forward_vgg_new'

I have since read from the documentation that it is better for me to save the model's state_dict:

            state = {
                    'net':net.state_dict(),
                    'acc':acc,
                    'epoch':epoch,
            }
...
            torch.save(state, ...)

While I will change the code for future runs, I was wondering if it was possible to salvage the models I have already trained. I naively already tried to import the VGG class and add my forward_vgg_new method to it:

setattr(VGG, 'forward_vgg_new', forward_vgg_new)

before calling torch.load, but it doesn't work.

Output: To solve the problem, I went directly into the VGG library and temporarily added my function so that I could load the saved models and save only their state dicts. I reverted the changes to the VGG library after I recovered the saves. Not the most graceful way of fixing the problem, but it worked.


Instruction: Pytorch Multiclass Logistic Regression Type Errors

Input:
I'm new to ML and even more naive with Pytorch. Here's the problem. (I've skipped certain parts like the random_split() which seem to work just fine)

I've to predict wine quality (red) which from the dataset is the last column with 6 classes

That's what my dataset looks like

The link to the dataset (winequality-red.csv)

features = df.drop(['quality'], axis = 1)
targets = df.iloc[:, -1] # theres 6 classes

dataset = TensorDataset(torch.Tensor(np.array(features)).float(), torch.Tensor(targets).float())
# here's where I think the error might be, but I might be wrong

batch_size = 8
# Dataloader

train_loader = DataLoader(train_ds, batch_size, shuffle = True)
val_loader = DataLoader(val_ds, batch_size)
test_ds = DataLoader(test_ds, batch_size)

input_size = len(df.columns) - 1
output_size = 6
threshold = .5

class WineModel(nn.Module):

  def __init__(self):
    super().__init__()
    self.linear = nn.Linear(input_size, output_size)

  def forward(self, xb):
    out = self.linear(xb)
    return out

model = WineModel()
n_iters = 2000
num_epochs = n_iters / (len(train_ds) / batch_size)
num_epochs = int(num_epochs)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)

# the part below returns the error on running
iter = 0
for epoch in range(num_epochs):
  for i, (x, y) in enumerate(train_loader):
    optimizer.zero_grad()
    outputs = model(x)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()

RuntimeError: expected scalar type Long but found Float

Hopefully that is sufficient info

Output:
The targets for nn.CrossEntropyLoss are given as the class indices, which are required to be integers, to be precise they need to be of type torch.long, which is equivalent to torch.int64.

You converted the targets to floats, but you should convert them to longs:

dataset = TensorDataset(torch.Tensor(np.array(features)).float(), torch.Tensor(targets).long())

Since the targets are the indices of the classes, they must be in range [0, num_classes - 1]. As you have 6 classes that would be in range [0, 5]. Having a quick look at your data, the quality uses values in range [3, 8]. Even though you have 6 classes, the values cannot be used directly as the classes. If you list the classes as classes = [3, 4, 5, 6, 7, 8], you can see that the first class is 3, classes[0] == 3, up to the last class being classes[5] == 8.

You need to replace the class values with the indices, just like you would for named classes (e.g. if you had the classes dog and cat, dog would be 0 and cat would be 1), but you can avoid having to look them up, since the values are simply shifted by 3, i.e. index = classes[index] - 3. Therefore you can subtract 3 from the entire target tensor:

torch.Tensor(targets).long() - 3


Instruction: Cross Entropy Calculation in PyTorch tutorial

Input:
I'm reading the Pytorch tutorial of a multi-class classification problem. And I find the behavior of Loss calculation in Pytorch confuses me a lot. Can you help me with this?

The model used for classification goes like this:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The training process goes as follows:

optimizer.zero_grad()
outputs = net(inputs)
loss = nn.CrossEntropyLoss(outputs, labels)
loss.backward()
optimizer.step()

My question is: What's the exact behavior of Loss calculation in Pytorch here? During each iteration, the input of nn.CrossEntropyLoss() has two parts:

The output of the model, which is a 10 by 1 tensor, with different values in it. This is a tensor without normalized into probability.
The label as a scalar, like 1 or 2 or 3.

As far as I know, the calculation of cross-entropy usually used between two tensors like:

Target as [0,0,0,1], where 1 is the right class
Output tensor as [0.1,0.2,0.3,0.4], where the sum as 1.

So based on this assumption, nn.CrossEntropyLoss() here needs to achieve:

Firstly normalize the output tensor into possibility one.
Encode the label into one-hot ones, like 2 in 5 class as [0,1,0,0,0]. The length must be the same as output tensor.
Then calculate the loss.

May I ask is this what nn.CrossEntropyLoss() does? Or do we need to one-hot encoding the true label before we input into the model?

Thank you a lot for your time in advance!

Output:
nn.CrossEntropyLoss first applies log-softmax (log(Softmax(x)) to get log probabilities and then calculates the negative-log likelihood as mentioned in the documentation:

  This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.

When using one-hot encoded targets, the cross-entropy can be calculated as follows:

where y is the one-hot encoded target vector and ŷ is the vector of probabilities for each class. To get the probabilities you would apply softmax to the output of the model. The logarithm of the probabilities is used, and PyTorch just combines the logarithm and the softmax into one operation nn.LogSoftmax(), for numerical stability.

Since all of the values except one in the one-hot vector are zero, only a single term of the sum will be non-zero. Therefore given the actual class, it can be simplified to:

As long as you know the class index, the loss can be calculated directly, making it more efficient than using a one-hot encoded target, hence nn.CrossEntropyLoss expects the class indices.

The full calculation is given in the documentation of nn.CrossEntropyLoss:

  The loss can be described as:


Instruction: PyTorch: The number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1)

Input:
I'm trying to convert a CPU model to GPU using Pytorch, but I'm running into issues. I'm running this on Colab and I'm sure that Pytorch detects a GPU. This is a deep Q network (RL).

I declare my network as: Q = Q_Network(input_size, hidden_size, output_size).to(device)

I ran into an issue when I tried to pass arguments through the network (It expected type cuda but got type cpu) so I add .to(device):

batch = np.array(shuffled_memory[i:i+batch_size])
b_pobs = np.array(batch[:, 0].tolist(), dtype=np.float32).reshape(batch_size, -1)
b_pact = np.array(batch[:, 1].tolist(), dtype=np.int32)
b_reward = np.array(batch[:, 2].tolist(), dtype=np.int32)
b_obs = np.array(batch[:, 3].tolist(), dtype=np.float32).reshape(batch_size, -1)
b_done = np.array(batch[:, 4].tolist(), dtype=np.bool)

q = Q(torch.from_numpy(b_pobs).to(device))
q_ = Q_ast(torch.from_numpy(b_obs).to(device))

maxq = torch.max(q_.data,axis=1)
target = copy.deepcopy(q.data)

for j in range(batch_size):
    print(target[j, b_pact[j]].shape) # torch.Size([])
    target[j, b_pact[j]] = b_reward[j]+gamma*maxq[j]*(not b_done[j]) #I run into issues here

Here is the error:

RuntimeError: expand(torch.cuda.FloatTensor{[50]}, size=[]): the number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1)

Output:
target[j, b_pact[j]] is a single element of the tensor (a scalar, hence size of torch.Size([])). If you want to assign anything to it, the right hand side can only be a scalar. That is not the case, as one of the terms is a tensor with 1 dimension (a vector), namely your maxq[j].

When specifying a dimension dim (axis is treated as a synonym) to torch.max, it will return a named tuple of (values, indices), where values contains the maximum values and indices the location of each of the maximum values (equivalent to argmax).

maxq[j] is not indexing into the maximum values, but rather the tuple of (values, indices). If you only want the values you can use one of the following to get the values out of the tuple (all of them are equivalent, you can use whichever you prefer):

# Destructure/unpack and ignore the indices
maxq, _ = torch.max(q_.data,axis=1)

# Access first element of the tuple
maxq = torch.max(q_.data,axis=1)[0]

# Access `values` of the named tuple
maxq  = torch.max(q_.data,axis=1).values


Instruction: padding='same' conversion to PyTorch padding=#

Input:
I'm trying to convert the following Keras model code to pytorch, but am having problems dealing with padding='same'.

    model = Sequential()
    model.add(Conv2D(64, (3, 3), input_shape=img_size))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(Dropout(0.3))
    model.add(Conv2D(64, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

Which produces the following summary:

Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 30, 30, 64)        1792
_________________________________________________________________
batch_normalization_1 (Batch (None, 30, 30, 64)        120
_________________________________________________________________
activation_1 (Activation)    (None, 30, 30, 64)        0
_________________________________________________________________
dropout_1 (Dropout)          (None, 30, 30, 64)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 30, 30, 64)        36928
_________________________________________________________________
batch_normalization_2 (Batch (None, 30, 30, 64)        120
_________________________________________________________________
activation_2 (Activation)    (None, 30, 30, 64)        0
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 15, 15, 64)        0
=================================================================
Total params: 38,960
Trainable params: 38,840
Non-trainable params: 120

Right now, I would write:

self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3,
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Conv2d(64, 64, kernel_size=3, padding = ?
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2, padding = ?),
        )

Where padding should have numerical value. I was wondering if there is an easier way to calculate this since we're using padding='same'.

Also, the next line of the Keras model looks like:

model.add(Conv2D(128, (3, 3), padding='same'))

So I really need to brush up on how to calculate padding, especially after stride too. From a rough eye only, is the padding 2?

Output:
W:input volume size

F:kernel size

S:stride

P:amount of padding

size of output volume = (W-F+2P)/S+1

e.g.

input:7x7, kernel:3x3, stride:1, pad:0

output size = (7-3+2*0)/1+1 = 5 =>5x5


Instruction: pytorch affine_grid: what is the theta input?

Input:
When trying to use torch.nn.functional.affine_grid, it requires a theta affine matrix of size (N x 3 x 4) according to the documentation. I thought a general affine matrix is (N x 4 x 4). What is the supposed affine matrix format in pytorch?

An example of 3D rotation affine input would be ideal. Appreciate your help.

Output:
The dimensions you mention are applicable for the case of 3D inputs, that is you wish to apply 3D geometric transforms on the input tensor x of shape bxcxdxhxw.
A transformation to points in 3D (represented as 4-vector in homogeneous coordinates as (x, y, z, 1)) should be, in the general case, a 4x4 matrix as you noted.
However, since we restrict ourselves to homogeneous coordinates, i.e., the fourth coordinate must be 1, the 4th row of the matrix must be (0, 0, 0, 1) (see this).
Therefore, there's no need to explicitly code this last row.

To conclude, a 3D transformation composed of a 3x3 rotation R and 3d translation t is simply the 3x4 matrix:

theta = [R t]


Instruction: How to balance the generator and the discriminator performances in a GAN?

Input:
It's the first time I'm working with GANs and I am facing an issue regarding the Discriminator repeatedly outperforming the Generator. I am trying to reproduce the PA model from this article and I'm looking at this slightly different implementation to help me out.

I have read quite a lot of papers on how GANs work and also followed some tutorials to understand them better. Moreover, I've read articles on how to overcome the major instabilities, but I can't find a way to overcome this behavior.

In my environment, I'm using PyTorch and BCELoss(). Following the DCGAN PyTorch tutorial, I'm using the following training loop:

criterion = nn.BCELoss()
train_d = False
# Discriminator true
optim_d.zero_grad()
disc_train_real = target.to(device)
batch_size = disc_train_real.size(0)
label = torch.full((batch_size,), 1, device=device).cuda()
output_d = discriminator(disc_train_real).view(-1)
loss_d_real = criterion(output_d, label).cuda()
if lossT:
    loss_d_real *= 2
if loss_d_real.item() > 0.3:
    loss_d_real.backward()
    train_d = True
D_x = output_d.mean().item()
# Discriminator false
output_g = generator(image)
output_d = discriminator(output_g.detach()).view(-1)
label.fill_(0)
loss_d_fake = criterion(output_d, label).cuda()
D_G_z1 = output_d.mean().item()
if lossT:
    loss_d_fake *= 2
loss_d = loss_d_real + loss_d_fake
if loss_d_fake.item() > 0.3:
    loss_d_fake.backward()
    train_d = True
if train_d:
    optim_d.step()

# Generator
label.fill_(1)
output_d = discriminator(output_g).view(-1)
loss_g = criterion(output_d, label).cuda()
D_G_z2 = output_d.mean().item()
if lossT:
    loss_g *= 2

loss_g.backward()
optim_g.step()

and, after a period of settlement, everything seems to work fine:

Epoch 1/5 - Step: 1900/9338  Loss G: 3.057388  Loss D: 0.214545  D(x): 0.940985  D(G(z)): 0.114064 / 0.114064
Time for the last step: 51.55 s    Epoch ETA: 01:04:13
Epoch 1/5 - Step: 2000/9338  Loss G: 2.984724  Loss D: 0.222931  D(x): 0.879338  D(G(z)): 0.159163 / 0.159163
Time for the last step: 52.68 s    Epoch ETA: 01:03:24
Epoch 1/5 - Step: 2100/9338  Loss G: 2.824713  Loss D: 0.241953  D(x): 0.905837  D(G(z)): 0.110231 / 0.110231
Time for the last step: 50.91 s    Epoch ETA: 01:02:29
Epoch 1/5 - Step: 2200/9338  Loss G: 2.807455  Loss D: 0.252808  D(x): 0.908131  D(G(z)): 0.218515 / 0.218515
Time for the last step: 51.72 s    Epoch ETA: 01:01:37
Epoch 1/5 - Step: 2300/9338  Loss G: 2.470529  Loss D: 0.569696  D(x): 0.620966  D(G(z)): 0.512615 / 0.350175
Time for the last step: 51.96 s    Epoch ETA: 01:00:46
Epoch 1/5 - Step: 2400/9338  Loss G: 2.148863  Loss D: 1.071563  D(x): 0.809529  D(G(z)): 0.114487 / 0.114487
Time for the last step: 51.59 s    Epoch ETA: 00:59:53
Epoch 1/5 - Step: 2500/9338  Loss G: 2.016863  Loss D: 0.904711  D(x): 0.621433  D(G(z)): 0.440721 / 0.435932
Time for the last step: 52.03 s    Epoch ETA: 00:59:02
Epoch 1/5 - Step: 2600/9338  Loss G: 2.495639  Loss D: 0.949308  D(x): 0.671085  D(G(z)): 0.557924 / 0.420826
Time for the last step: 52.66 s    Epoch ETA: 00:58:12
Epoch 1/5 - Step: 2700/9338  Loss G: 2.519842  Loss D: 0.798667  D(x): 0.775738  D(G(z)): 0.246357 / 0.265839
Time for the last step: 51.20 s    Epoch ETA: 00:57:19
Epoch 1/5 - Step: 2800/9338  Loss G: 2.545630  Loss D: 0.756449  D(x): 0.895455  D(G(z)): 0.403628 / 0.301851
Time for the last step: 51.88 s    Epoch ETA: 00:56:27
Epoch 1/5 - Step: 2900/9338  Loss G: 2.458109  Loss D: 0.653513  D(x): 0.820105  D(G(z)): 0.379199 / 0.103250
Time for the last step: 53.50 s    Epoch ETA: 00:55:39
Epoch 1/5 - Step: 3000/9338  Loss G: 2.030103  Loss D: 0.948208  D(x): 0.445385  D(G(z)): 0.303225 / 0.263652
Time for the last step: 51.57 s    Epoch ETA: 00:54:47
Epoch 1/5 - Step: 3100/9338  Loss G: 1.721604  Loss D: 0.949721  D(x): 0.365646  D(G(z)): 0.090072 / 0.232912
Time for the last step: 52.19 s    Epoch ETA: 00:53:55
Epoch 1/5 - Step: 3200/9338  Loss G: 1.438854  Loss D: 1.142182  D(x): 0.768163  D(G(z)): 0.321164 / 0.237878
Time for the last step: 50.79 s    Epoch ETA: 00:53:01
Epoch 1/5 - Step: 3300/9338  Loss G: 1.924418  Loss D: 0.923860  D(x): 0.729981  D(G(z)): 0.354812 / 0.318090
Time for the last step: 52.59 s    Epoch ETA: 00:52:11

that is, the gradients on the Generator are higher and start to decrease after a while, and in the meanwhile the gradients on the Discriminator rise up. As for the losses, the Generator goes down while the Discriminator goes up. If compared to the tutorial, I guess this can be acceptable.

Here's my first question: I've noticed that on the tutorial (usually) as D_G_z1 rises, D_G_z2 decreases (and viceversa), while in my example this happens a lot less. Is it just a coincidence or am I doing something wrong?

Given that, I've let the training procedure go on, but now I'm noticing this:

Epoch 3/5 - Step: 1100/9338  Loss G: 4.071329  Loss D: 0.031608  D(x): 0.999969  D(G(z)): 0.024329 / 0.024329
Time for the last step: 51.41 s    Epoch ETA: 01:11:24
Epoch 3/5 - Step: 1200/9338  Loss G: 3.883331  Loss D: 0.036354  D(x): 0.999993  D(G(z)): 0.043874 / 0.043874
Time for the last step: 51.63 s    Epoch ETA: 01:10:29
Epoch 3/5 - Step: 1300/9338  Loss G: 3.468963  Loss D: 0.054542  D(x): 0.999972  D(G(z)): 0.050145 / 0.050145
Time for the last step: 52.47 s    Epoch ETA: 01:09:40
Epoch 3/5 - Step: 1400/9338  Loss G: 3.504971  Loss D: 0.053683  D(x): 0.999972  D(G(z)): 0.052180 / 0.052180
Time for the last step: 50.75 s    Epoch ETA: 01:08:41
Epoch 3/5 - Step: 1500/9338  Loss G: 3.437765  Loss D: 0.056286  D(x): 0.999941  D(G(z)): 0.058839 / 0.058839
Time for the last step: 52.20 s    Epoch ETA: 01:07:50
Epoch 3/5 - Step: 1600/9338  Loss G: 3.369209  Loss D: 0.062133  D(x): 0.955688  D(G(z)): 0.058773 / 0.058773
Time for the last step: 51.05 s    Epoch ETA: 01:06:54
Epoch 3/5 - Step: 1700/9338  Loss G: 3.290109  Loss D: 0.065704  D(x): 0.999975  D(G(z)): 0.056583 / 0.056583
Time for the last step: 51.27 s    Epoch ETA: 01:06:00
Epoch 3/5 - Step: 1800/9338  Loss G: 3.286248  Loss D: 0.067969  D(x): 0.993238  D(G(z)): 0.063815 / 0.063815
Time for the last step: 52.28 s    Epoch ETA: 01:05:09
Epoch 3/5 - Step: 1900/9338  Loss G: 3.263996  Loss D: 0.065335  D(x): 0.980270  D(G(z)): 0.037717 / 0.037717
Time for the last step: 51.59 s    Epoch ETA: 01:04:16
Epoch 3/5 - Step: 2000/9338  Loss G: 3.293503  Loss D: 0.065291  D(x): 0.999873  D(G(z)): 0.070188 / 0.070188
Time for the last step: 51.85 s    Epoch ETA: 01:03:25
Epoch 3/5 - Step: 2100/9338  Loss G: 3.184164  Loss D: 0.070931  D(x): 0.999971  D(G(z)): 0.059657 / 0.059657
Time for the last step: 52.14 s    Epoch ETA: 01:02:34
Epoch 3/5 - Step: 2200/9338  Loss G: 3.116310  Loss D: 0.080597  D(x): 0.999850  D(G(z)): 0.074931 / 0.074931
Time for the last step: 51.85 s    Epoch ETA: 01:01:42
Epoch 3/5 - Step: 2300/9338  Loss G: 3.142180  Loss D: 0.073999  D(x): 0.995546  D(G(z)): 0.054752 / 0.054752
Time for the last step: 51.76 s    Epoch ETA: 01:00:50
Epoch 3/5 - Step: 2400/9338  Loss G: 3.185711  Loss D: 0.072601  D(x): 0.999992  D(G(z)): 0.076053 / 0.076053
Time for the last step: 50.53 s    Epoch ETA: 00:59:54
Epoch 3/5 - Step: 2500/9338  Loss G: 3.027437  Loss D: 0.083906  D(x): 0.997390  D(G(z)): 0.082501 / 0.082501
Time for the last step: 52.06 s    Epoch ETA: 00:59:03
Epoch 3/5 - Step: 2600/9338  Loss G: 3.052374  Loss D: 0.085030  D(x): 0.999924  D(G(z)): 0.073295 / 0.073295
Time for the last step: 52.37 s    Epoch ETA: 00:58:12

not only D(x) has increased again and it's stuck to almost one, but also both D_G_z1 and D_G_z2 always show the same value. Moreover, looking at the losses it seems pretty clear that the Discriminator has outperformed the Generator. This behavior has gone on and on for the rest of the epoch and for all the next one, until the end of the training.

Hence my second question: is this normal? If not, what am I doing wrong within the procedure? How can I achieve a more stable training?

EDIT: I've tried to train the network using the MSELoss() as suggested and this is the output:

Epoch 1/1 - Step: 100/9338  Loss G: 0.800785  Loss D: 0.404525  D(x): 0.844653  D(G(z)): 0.030439 / 0.016316
Time for the last step: 55.22 s    Epoch ETA: 01:25:01
Epoch 1/1 - Step: 200/9338  Loss G: 1.196659  Loss D: 0.014051  D(x): 0.999970  D(G(z)): 0.006543 / 0.006500
Time for the last step: 51.41 s    Epoch ETA: 01:21:11
Epoch 1/1 - Step: 300/9338  Loss G: 1.197319  Loss D: 0.000806  D(x): 0.999431  D(G(z)): 0.004821 / 0.004724
Time for the last step: 51.79 s    Epoch ETA: 01:19:32
Epoch 1/1 - Step: 400/9338  Loss G: 1.198960  Loss D: 0.000720  D(x): 0.999612  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.47 s    Epoch ETA: 01:18:09
Epoch 1/1 - Step: 500/9338  Loss G: 1.212810  Loss D: 0.000021  D(x): 0.999938  D(G(z)): 0.000000 / 0.000000
Time for the last step: 52.18 s    Epoch ETA: 01:17:11
Epoch 1/1 - Step: 600/9338  Loss G: 1.216168  Loss D: 0.000000  D(x): 0.999945  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.24 s    Epoch ETA: 01:16:02
Epoch 1/1 - Step: 700/9338  Loss G: 1.212301  Loss D: 0.000000  D(x): 0.999970  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.61 s    Epoch ETA: 01:15:02
Epoch 1/1 - Step: 800/9338  Loss G: 1.214397  Loss D: 0.000005  D(x): 0.999973  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.58 s    Epoch ETA: 01:14:04
Epoch 1/1 - Step: 900/9338  Loss G: 1.212016  Loss D: 0.000003  D(x): 0.999932  D(G(z)): 0.000000 / 0.000000
Time for the last step: 52.20 s    Epoch ETA: 01:13:13
Epoch 1/1 - Step: 1000/9338  Loss G: 1.215162  Loss D: 0.000000  D(x): 0.999988  D(G(z)): 0.000000 / 0.000000
Time for the last step: 52.28 s    Epoch ETA: 01:12:23
Epoch 1/1 - Step: 1100/9338  Loss G: 1.216291  Loss D: 0.000000  D(x): 0.999983  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.78 s    Epoch ETA: 01:11:28
Epoch 1/1 - Step: 1200/9338  Loss G: 1.215526  Loss D: 0.000000  D(x): 0.999978  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.88 s    Epoch ETA: 01:10:35

As can be seen, the situation gets even worse. Moreover, reading the EnhanceNet paper all over again, Section 4.2.4 (Adversarial Training) states that the adversarial loss function used is a BCELoss(), as I would expect to solve the vanishing gradients problem that I get with MSELoss().

Output:
Interpreting GAN Losses are a bit of a black art because the actual loss values

Question 1: The frequency of swinging between a discriminator/generator dominance will vary based on a few factors primarily (in my experience): learning rates and batch sizes which will impact the propagated loss. The particular loss metrics used will impact variance in how the D & G networks train. The EnhanceNet paper (for baseline) and the tutorial use a Mean Squared Error loss too - you're using a Binary Cross Entropy loss which will change the rate at which the networks converge. I'm no expert so here's a pretty good link to Rohan Varma's article that explains the difference between loss functions. Would be curious to see if your network behaves differently when you change the loss function - try it and update the question?

Question 2: Over time both the D and G losses should settle to a value, however it's somewhat difficult to tell whether they've converged on strong performance or whether they've converged due to something like mode collapse/diminishing gradients (Jonathan Hui's explanation on problems in training GANs). The best way I've found is to actually inspect a cross section of the generated images and either visually inspect the output or use some kind of perceptual metrics (SSIM, PSNR, PIQ, etc.) across the generated image set.

Some other useful leads that you might find useful in finding an ans:

This post has a couple of reasonably good pointers on interpreting GAN Losses.

Ian Goodfellow's NIPS2016 tutorial also has some solid ideas on how to balance D & G training.


Instruction: How to create new pandas dataframe column containing values of all other columns as a tensor?

Input:
I'm trying to create a new feature column which is a tensor containing the values in the existing columns. So if col A's value is '1', col B's value is '0', and col C's value is 0, then the new feature column's value will be [1,0,0].
I tried the following code:
import numpy as np
import pandas as pd
import torch

df = pd.DataFrame({"A":[1,1,0], "B":[0,1,1], "Sentiment":[0,0,1]})

df["new_feature"] = [df["A"].values, df["B"].values, df["C"].values]

...but the result is not what I need. The result is getting the values down each column rather than the values across the row (multiple column values). For example, the new_feature column value for the first row should be [1,0,0] but its showing [1,1,0]
My ultimate aim is to get a dataframe column that I can use as a torch tensor to input into a neural net.

Output:
Use torch.from_numpy with apply lambda function.

df["new_feature"] = df.apply(lambda x:torch.from_numpy(x.to_numpy()), axis = 1)
df
   A  B  C                        new_feature
0  1  0  0  [tensor(1), tensor(0), tensor(0)]
1  1  1  0  [tensor(1), tensor(1), tensor(0)]
2  0  1  1  [tensor(0), tensor(1), tensor(1)]

df["new_feature"][0]
tensor([1, 0, 0])

First convert dataframe values to numpy array using pd.Series.to_numpy and
then convert numpy array to tensor using torch.from_numpy.


Instruction: Recommended way to replace several values in a tensor at once?

Input:
Is there a batch way to replace several particular values in a pytorch tensor at once without a for loop?

Example:

old_values = torch.Tensor([1, 2, 3, 4, 5, 5, 2, 3, 3, 2])
old_new_value = [[2,22], [3,33], [6, 66]]

old_new_value = [[2,22], [3,33], [6, 66]], which means 2 should be replaced by 22, and 3 should be replaced by 33 and 6 to 66

Can I have an efficient way to achieve the following end_result?

end_result = torch.Tensor([1, 22, 33, 4, 5, 5, 22, 33, 33, 22])

Note that old_values is not unique. Also, it is possible that old_new_value has a pair here(6, 66) that does not exist in the old_values.
Also, the old_new_values includes unique rows,

Output:
If you don't have any duplicate elements in your input tensor, here's one straightforward way using masking and value assignment using basic indexing. (I'll assume that the data type of the input tensor is int. But, you can simply adapt this code in a straightforward manner to other dtypes). Below is a reproducible illustration, with explanations interspersed in inline comments.

# input tensors to work with
In [75]: old_values
Out[75]: tensor([1, 2, 3, 4, 5], dtype=torch.int32)

In [77]: old_new_value
Out[77]:
tensor([[ 2, 22],
        [ 3, 33]], dtype=torch.int32)

# generate a boolean mask using the values that need to be replaced (i.e. 2 & 3)
In [78]: boolean_mask = (old_values == old_new_value[:, :1]).sum(dim=0).bool()

In [79]: boolean_mask
Out[79]: tensor([False,  True,  True, False, False])

# assign the new values by basic indexing
In [80]: old_values[boolean_mask] = old_new_value[:, 1:].squeeze()

# sanity check!
In [81]: old_values
Out[81]: tensor([ 1, 22, 33,  4,  5], dtype=torch.int32)

A small note on efficiency: Throughout the whole process, we never made any copy of the data (i.e. we operate only on new views by massaging the shapes according to our needs). Therefore, the runtime would be blazing fast.


Instruction: what is the pytorch's view equivalence with tensorflow 2.0?

Input:
l_conv7 = self.loc_conv7(conv7_feats)  # (N, 24, 19, 19)
l_conv7 = l_conv7.permute(0, 2, 3, 1).contiguous()  # (N, 19, 19, 24)
l_conv7 = l_conv7.view(batch_size, -1, 4)  # (N, 2166, 4), there are a total 2116 boxes on this feature map

what is the equivalence with torch's view in TensorFlow?
how to change the l_conv7.view in TensorFlow 2.0?

Output:
Use

l_conv7.reshape(batch_size, -1, 4)


Instruction: Weight Normalization in PyTorch

Input:
An important weight normalization technique was introduced in this paper and has been included in PyTorch since long as follows:

    from torch.nn.utils import weight_norm
    weight_norm(nn.Conv2d(in_channles, out_channels))

From the docs I get to know, weight_norm does re-parametrization before each forward() pass. But I am not sure if this re-parameterization is also happening during the inference when everything is running inside with torch.no_grad() and the model is set to eval() mode.

Can someone please help me know if weight_norm is active only during training or during the inference mode as described above?

Thank you

Output:
I have finally figured out the problem.

Batch normalization learns two parameters during training and uses them for inference. Thus it is necessary to change its behaviour using eval() to tell not to modify them any further.

I then scrutinizingly checked the weight normalization paper and found it to be 'inherently deterministic'. It simply decouples the original weight vectors as product of two quantities as shown below.

w = g . v

Obviously either you use LHS for computing output or RHS it does not matter. However by decoupling it into two vectors and passing them to optimizer and deleting the w parameter better training is achieved. For reasons refer the paper where things are nicely described.

Thus it does not matter if weight normalization is removed or not during testing. To validate this I tried the following small code.

import torch
import torch.nn as nn
from torch.nn.utils import weight_norm as wn
from torch.nn.utils import remove_weight_norm as wnr

# define the model 'm'
m = wn(nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, padding=1, bias=True))

ip = torch.rand(1,1,5,5)
target = torch.rand(1,1,5,5)
l1 = torch.nn.L1Loss()
optimizer = torch.optim.Adam(m.parameters())

# begin training
for _ in range(5):
    out = m(ip)
    loss = l1(out,target)
    loss.backward()
    optimizer.step()

with torch.no_grad():
    m.eval()
    print('\no/p after training with wn: {}'.format(m(ip)))
    wnr(m)
    print('\no/p after training without wn: {}'.format(m(ip)))

# begin testing
m2 = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3,padding=1, bias=True)
m2.load_state_dict(m.state_dict())

with torch.no_grad():
    m2.eval()
    out = m2(ip)
    print('\nOutput during testing and without weight_norm: {}'.format(out))

And the output is below,

o/p after training with wn:
tensor([[[[0.0509, 0.3286, 0.4612, 0.1795, 0.0307],
          [0.1846, 0.3931, 0.5713, 0.2909, 0.4026],
          [0.1716, 0.5971, 0.4297, 0.0845, 0.6172],
          [0.2938, 0.2389, 0.4478, 0.5828, 0.6276],
          [0.1423, 0.2065, 0.5024, 0.3979, 0.3127]]]])

o/p after training without wn:
tensor([[[[0.0509, 0.3286, 0.4612, 0.1795, 0.0307],
          [0.1846, 0.3931, 0.5713, 0.2909, 0.4026],
          [0.1716, 0.5971, 0.4297, 0.0845, 0.6172],
          [0.2938, 0.2389, 0.4478, 0.5828, 0.6276],
          [0.1423, 0.2065, 0.5024, 0.3979, 0.3127]]]])

Output during testing and without weight_norm:
tensor([[[[0.0509, 0.3286, 0.4612, 0.1795, 0.0307],
          [0.1846, 0.3931, 0.5713, 0.2909, 0.4026],
          [0.1716, 0.5971, 0.4297, 0.0845, 0.6172],
          [0.2938, 0.2389, 0.4478, 0.5828, 0.6276],
          [0.1423, 0.2065, 0.5024, 0.3979, 0.3127]]]])

Please see that all the values are exactly same as only reparameterization is happening.

Regarding,

  Then I tested two models using C++ code with libtorch. But the results are not the same.

See https://github.com/pytorch/pytorch/issues/21275 which reports a bug with TorchScript.

And regarding,

  I am wondering what does weight_norm do in inference? Is it usefull?

The answer is it does nothing. you do x * 2 or x * (1+1) does not matter. It is not useful but not harmful either. So better remove it.


Instruction: Expected object of scalar type Float but got scalar type Long for argument #2 'mat1' in call to _th_addmm

Input:
The error Expected object of scalar type Float but got scalar type Long for argument #2 'mat1' in call to _th_addmm is being displayed after running the code below.

import numpy as np
import pandas as pd
import nltk
from nltk.stem import WordNetLemmatizer
nltk.download('wordnet')
import re
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader, TensorDataset

... (created a pandas dataframe containing a 'tweet', 'sentiment' and 47 one-hot bag of words cols)

# create train_target data set containing just the target
train_target = torch.tensor(df_train['sentiment'].values, dtype=torch.long)

# create train predictor features data set containing just the predictor features
train = torch.tensor(df_train.drop(['tweet','sentiment'], axis = 1).values, dtype=torch.long)

# convert to torch tensor and define data loader (train)
train_tensor = TensorDataset(train, train_target)
trainset = torch.utils.data.DataLoader(train_tensor, batch_size=2, shuffle=False)

# create test_target data set containing just the target
test_target = torch.tensor(df_test['sentiment'].values, dtype=torch.long)

# create test predictor features data set containing just the predictor features
test = torch.tensor(df_test.drop(['tweet','sentiment'], axis = 1).values, dtype=torch.long)

# convert to torch tensor and define data loader (test)
test_tensor = TensorDataset(test, test_target)
testset = torch.utils.data.DataLoader(test_tensor, batch_size=2, shuffle=False)

input_length = 47

class Net(nn.Module):   # create a new class called Net that inherits from nn's "Module" class

    # initialise our Net class
    def __init__(self):
        super().__init__()    # run the initialisation function of the nn.Module class (the parent class)
        self.fc1 = nn.Linear(input_length, 768)   # define the first fully connected layer.
        self.fc2 = nn.Linear(768, 768)    # define the 2nd layer
        self.fc3 = nn.Linear(768, 2)    # output layer. 2 classes so output is size 2

    # define how data will flow through feed-forward network
    def forward(self, x):
        x = F.relu(self.fc1(x))  # x becomes the output of running the 1st layer, after relu activation is applied
        x = F.relu(self.fc2(x))    # x then becomes the output of the 2nd layer...
        x = self.fc3(x)

        return F.log_softmax(x, dim=1)  # we want a probability distribution across the 2 classes so using softmax
                                        # dim=1 because we want the probabilities across the classes, not
                                        # the batches

# create a Net object called net
net = Net()
print(net)

# Train the model (net)
optimiser = optim.Adam(net.parameters(),  # net.parameters is everything thats adjustable in our model
                      lr=0.001)  # learning rate
epochs = 10
for epoch in range(epochs):
    for batch_data in trainset:
        X, y = batch_data  # set X as the input and y as the label
        net.zero_grad() # start with gradients of zero
        output = net(X.view(-1, input_length))  # run the model (could put batch size in instead of -1)
        loss = F.nll_loss(output, y)   # how wrong were we? Calculate loss.
        loss.backward()    # backpropagate the loss (how much did each weight contribute to the loss?)
        optimiser.step()    # adjust the weights based on the backpropagation
    print('Epoch: ', epoch, ' Loss: ', loss)

What do I need to do to fix this? I'm trying to create a neural network to do sentiment classification on some text data from tweets.

Output:
The inputs to the nn.Linear layers, and therefore your model, need to be floats, not longs.

You need to change the features to use dtype=torch.float:

train = torch.tensor(df_train.drop(['tweet','sentiment'], axis = 1).values, dtype=torch.float)

test = torch.tensor(df_test.drop(['tweet','sentiment'], axis = 1).values, dtype=torch.float)

That only applies to the input features, whereas your targets need to remain torch.long for the NLL loss.


Instruction: How to attach hooks to ReLUs in Inception V3 from torchvision

Input:
I am using Inception v3 from torchvision. I tried to find the ReLUs within the model:

def recursively_find_submodules(model, submodule_type):
    module_list = []
    q = [model]
    while q:
        child = q.pop()
        if isinstance(child, submodule_type):
            module_list.append(child)
        q.extend(list(child.children()))
    return module_list

inception = torch.hub.load('pytorch/vision:v0.6.0', 'inception_v3', pretrained=True)
l = recursively_find_submodules(inception, torch.nn.ReLU) # l is empty!

So the ReLUs are not children of any module within the torch model. Upon closer inspection I found the ReLUs in the source code of torchvision but not as modules. In inception.py I found the following:

class BasicConv2d(nn.Module):

    def __init__(self, in_channels, out_channels, **kwargs):
        super(BasicConv2d, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs)
        self.bn = nn.BatchNorm2d(out_channels, eps=0.001)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        return F.relu(x, inplace=True)

So the BasicConv2d module uses the ReLU function to clamp it's output instead of the module (torch.nn.ReLU). I guess there is no way to hook up to ReLU functions and modify their input / output without modifying the whole model to use ReLU modules or is there a way to do this?

Output: You can hook to the batch-norm layer preceding the ReLU and attach there, taking into account you observe the inputs to the ReLU rather that the features after the activation.


Instruction: Why doesn't pytorch allow inplace operations on leaf variables?

Input:
So if I run this code in Pytorch:

x = torch.ones(2,2, requires_grad=True)
x.add_(1)

I will get the error:

  RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

I understand that Pytorch does not allow inplace operations on leaf variables and I also know that there are ways to get around this restrictions. What I don't understand is the philosophy behind this rule. Why is it wrong to change a leaf variable with inplace operations?

Output:
As I understand it, any time you do a non-traditional operation on a tensor that was initialized with requires_grad=True, Pytorch throws an error to make sure it was intentional. For example, you normally would only update a weight tensor using optimizer.step().
For another example, I ran into this issue when trying to update the values in a backprop-able tensor during network initialization.
self.weight_layer = nn.Parameter(data=torch.zeros(seq_length), requires_grad=True)
self.weight_layer[true_ids == 1] = -1.2

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

The problem is that, because requires_grad=True, the network doesn't know that I'm still initializing the values. If this is what you are trying to do, wrapping the update in a torch.no_grad block is one solution:
with torch.no_grad()
    self.weight_layer = nn.Parameter(data=torch.zeros(seq_length), requires_grad=True)
    self.weight_layer[true_ids == 1] = -1.2

Otherwise, you could just set requires_grad=True after you finish initializing the Tensor:
self.weight_layer = nn.Parameter(data=torch.zeros(seq_length))
self.weight_layer[true_ids == 1] = -1.2
self.weight_layer.requires_grad = True


Instruction: What is the best file type to save a 4D tensor?

Input:
I have some data I need to pre-process for a later step in a 3D Convolutional Network. The data comes in a file formatted like this:

   POSITION
x     y     z     (feature 1 x) (feature 1 y) (feature 1 z) (feature 2 x) (feature 2 y ...
1.2   0.54  2.3    0.04          0.2          -0.9           -0.2         0.65 ...
...(more rows of the same format)...

And after some other steps which involve operating on the positional data and the features, I get a pytorch tensor with dimensions [height][width][depth][features], or equivalently a numpy array, where the first three are positional data that I can use to plot the features using colours, and the [features] are vectors containing each of the feature values.

These are pretty large files and I'd like not have to perform the conversion from the first file format shown above to the tensor/array form later during processing. I'm thinking of using torch.save(tensor, 'file.pt').

My question is: what is the best file format to save this data so that it can be easily accessed later without the need for any pre-processing? Having to serialize it with PyTorch seems to be quite a convoluted way to save a type of data I would expect to have a more specific/designated file format.

Output:
I think I've found a more direct way to do it. Numpy supports saving its arrays as a .npy file.

The procedure is pretty straightforward. To save an array array_1 into the file numpy_array_1.npy, all you need to do is:

np.save('numpy_array_1.npy', array_l)

And then to load it into array_2:

array_2 = np.load('numpy_array_1.npy')


Instruction: `return_sequences = False` equivalent in pytorch LSTM

Input:
In tensorflow/keras, we can simply set return_sequences = False for the last LSTM layer before the classification/fully connected/activation (softmax/sigmoid) layer to get rid of the temporal dimension.

In PyTorch, I don't find anything similar. For the classification task, I don't need a sequence to sequence model but many to one architecture like this:

Here's my simple bi-LSTM model.

import torch
from torch import nn

class BiLSTMClassifier(nn.Module):
    def __init__(self):
        super(BiLSTMClassifier, self).__init__()
        self.embedding = torch.nn.Embedding(num_embeddings = 65000, embedding_dim = 64)
        self.bilstm = torch.nn.LSTM(input_size = 64, hidden_size = 8, num_layers = 2,
                                    batch_first = True, dropout = 0.2, bidirectional = True)
        # as we have 5 classes
        self.linear = nn.Linear(8*2*512, 5) # last dimension
    def forward(self, x):
        x = self.embedding(x)
        print(x.shape)
        x, _ = self.bilstm(x)
        print(x.shape)
        x = self.linear(x.reshape(x.shape[0], -1))
        print(x.shape)

# create our model

bilstmclassifier = BiLSTMClassifier()

If I observe the shapes after each layer,

xx = torch.tensor(X_encoded[0]).reshape(1,512)
print(xx.shape)
# torch.Size([1, 512])
bilstmclassifier(xx)
#torch.Size([1, 512, 64])
#torch.Size([1, 512, 16])
#torch.Size([1, 5])

What can I do so that the last LSTM returns a tensor with shape (1, 16) instead of (1, 512, 16)?

Output:
The simplest way to do this is by indexing into the tensor:

x = x[:, -1, :]

where x is the RNN output. Of course, if batch_first is False, one would have to use x[-1, :, :] (or just x[-1]) to index into the time axis instead. Turns out this is the same thing Tensorflow/Keras do. The relevant code can be found in K.rnn here:

last_output = tuple(o[-1] for o in outputs)

Note that the code at this point uses time_major data format, so the index is into the first axis. Also, outputs is a tuple because it can be multiple layers, state/cell pairs etc., but it is generally the sequence of outputs for all time steps.

This is then used in the RNN class as follows:

if self.return_sequences:
    output = K.maybe_convert_to_ragged(is_ragged_input, outputs, row_lengths)
else:
    output = last_output

So in total, we can see that return_sequences=False just uses outputs[-1].


Instruction: Pytorch: Index with tensor along multiple axes OR scatter to more than one index at once

Input:
I am trying to update very specific indices of a multidimensional tensor in Pytorch, and I am not sure how to access the correct indices. I can do this in a very straightforward way in Numpy:

import numpy as np
#set up the array containing the data
data = 100*np.ones((10,10,2))
data[5:,:,:] = 0
#select the data points that I want to update
idxs = np.nonzero(data.sum(2))
#generate the updates that I am going to do
updates = np.random.randint(5,size=(idxs[0].shape[0],2))
#update the data
data[idxs[0],idxs[1],:] = updates

I need to implement this in Pytorch but I am not sure how to do this. It seems like I need the scatter function but that only works along a single dimension instead of the multiple dimensions that I need. How can I do this?

Output:
These operations work exactly the same in their PyTorch counterparts, except for torch.nonzero, which by default returns a tensor of size [z, n] (where z is the number of non-zero elements and n the number of dimensions) instead of a tuple of n tensors with size [z] (as NumPy does), but that behaviour can be changed by setting as_tuple=True.

Other than that you can directly translate it to PyTorch, but you need to make sure that the types match, because you cannot assign a tensor of type torch.long (default of torch.randint) to a tensor of type torch.float (default of torch.ones). In this case, data is probably meant to have type torch.long:

#set up the array containing the data
data = 100*torch.ones((10,10,2), dtype=torch.long)
data[5:,:,:] = 0
#select the data points that I want to update
idxs = torch.nonzero(data.sum(2), as_tuple=True)
#generate the updates that I am going to do
updates = torch.randint(5,size=(idxs[0].shape[0],2))
#update the data
data[idxs[0],idxs[1],:] = updates


Instruction: pytorch custom dataset: DataLoader returns a list of tensors rather than tensor of a list

Input:
import torch

class Custom_Dataset(torch.utils.data.dataset.Dataset):
    def __init__(self, _dataset):
        self.dataset = _dataset

    def __getitem__(self, index):
        example, target = self.dataset[index]
        return example, target

    def __len__(self):
        return len(self.dataset)

train_data = [([1, 3, 5], 0),
              ([2, 4, 6], 1)]
train_loader = torch.utils.data.DataLoader(dataset=Custom_Dataset(train_data),
                                           batch_size=1,
                                           shuffle=False)

for inputs, targets in train_loader:
    print(inputs)
    print(targets)

I'm defining my training data as [([1, 3, 5], 0), ([2, 4, 6], 1)]: input([1, 3, 5]) paired target (0).

But when I fetch data from data loader, it becomes:

[tensor([1]), tensor([3]), tensor([5])]
tensor([0])

How do I get instead:

tensor([[1],
        [3],
        [5]])
tensor([0])

?

I know torch.stack can do the trick, but can I convert it in my custom dataset class?

Output:
One solution to get the desired input would be using numpy. Below I changed only two lines in your example to make it work.

import torch
import numpy as np

class Custom_Dataset(torch.utils.data.dataset.Dataset):
    def __init__(self, _dataset):
        self.dataset = _dataset

    def __getitem__(self, index):
        example, target = self.dataset[index]
        return np.array(example), target

    def __len__(self):
        return len(self.dataset)

train_data = [([1, 3, 5], 0),
              ([2, 4, 6], 1)]
train_loader = torch.utils.data.DataLoader(dataset=Custom_Dataset(train_data),
                                           batch_size=1,
                                           shuffle=False)

for inputs, targets in train_loader:
    print(inputs)
    print(targets)

Output of this code would be

tensor([[1, 3, 5]])
tensor([0])

tensor([[2, 4, 6]])
tensor([1])

But of course, I am assuming that having a row vector or a column vector does not make any difference to you. Otherwise, you might want to check this answer about transposing 1D vectors.

Hope this helps.


Instruction: Why use Caffe2 or Core-ML instead of LibTorch(.pt file) on iOS?

Input:
It seems like there are several ways to run Pytorch models on iOS.

PyTorch(.pt) -> onnx -> caffe2
PyTorch(.pt) -> onnx -> Core-ML (.mlmodel)
PyTorch(.pt) -> LibTorch (.pt)
PyTorch Mobile?

What is the difference between the above methods?
Why people use caffe2 or Core-ml (.mlmodel), which requires model format conversion, instead of LibTorch?

Output:
Core ML can use the Apple Neural Engine (ANE), which is much faster than running the model on the CPU or GPU. If a device has no ANE, Core ML can automatically fall back to the GPU or CPU.

I haven't really looked into PyTorch Mobile in detail, but I think it currently only runs on the CPU, not on the GPU. And it definitely won't run on the ANE because only Core ML can do that.

Converting models can be a hassle, especially from PyTorch which requires going through ONNX first. But you do end up with a much faster way to run those models.


Instruction: Get the Cross Entropy Loss in pytorch as in Keras

Input:
I am struggling to port a classification model form keras to pytorch. Especially the cross entropy loss seems to return totally different numbers.
import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])

print("Keras", K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred)))
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).argsort(dim=-1).float(), t.tensor(y_true).argmax(dim=-1)))```

prints:

Keras tf.Tensor([2.3369865], shape=(1,), dtype=float32)
PyTorch tensor(1.4587)

Since I have a custom loss function where cross entropy is a part of it, I would need to get similar if not the same numbers.

Output:
The problem is that they have different implementations.

As pytorch docs says, nn.CrossEntropyLoss combines nn.LogSoftmax() and nn.NLLLoss() in one single class. However, tensorflow docs specifies that keras.backend.categorical_crossentropy do not apply Softmax by default unless you set from_logits is True. For this reason, you should not use keras.backend.categorical_crossentropy without having previously apply softmax unless you use from_logits=True.

If you don't want to apply softmax beforehand you should use:

import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])

print("Keras", K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred), from_logits=True))
# output: Keras tf.Tensor([2.408051], shape=(1,), dtype=float32)
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).float(), t.tensor(y_true).argmax(dim=-1)))
# output: PyTorch tensor(2.4081)

Otherwise, you can apply Softmax manually before computing categorical_crossentropy

import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])

print("Keras", K.categorical_crossentropy(K.constant(y_true), K.softmax(K.constant(y_pred))))
# output: Keras tf.Tensor([2.408051], shape=(1,), dtype=float32)
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).float(), t.tensor(y_true).argmax(dim=-1)))
# output: PyTorch tensor(2.4081)

So you should not use keras.backend.categorical_crossentropy with from_logits=False as you were doing in your example.

  tf.keras.backend.categorical_crossentropy

  target:   A tensor of the same shape as output.

  output:   A tensor resulting from a softmax (unless from_logits is True, in which case output is expected to be the logits).

  from_logits:  Boolean, whether output is the result of a softmax, or is a tensor of logits.


Instruction: Pytorch ValueError: Target and input must have the same number of elements after change Image size

Input:
i have a working peace of code, which takes a Batchsize from 32 Image with the shape of 256*256 and i can train my neuronal network.

class Netz(nn.Module):
def __init__(self):
    super(Netz,self).__init__()
    self.conv1 = nn.Conv2d(3, 6, kernel_size=5)
    self.conv2 = nn.Conv2d(6, 12, kernel_size=3)
    self.conv3 = nn.Conv2d(12, 18, kernel_size=3)
    self.conv4 = nn.Conv2d(18, 24, kernel_size=3)
    self.fc1 = nn.Linear(4704, 1000)
    self.fc2 = nn.Linear(1000, 350)
    self.fc3 = nn.Linear(350,43)

def forward (self,x):
    x = F.relu(F.max_pool2d(self.conv1(x), 2))
    x = F.relu(F.max_pool2d(self.conv2(x), 2))
    x = F.relu(F.max_pool2d(self.conv3(x), 2))
    x = F.relu(F.max_pool2d(self.conv4(x), 2))
    x = x.view(-1,4704)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return torch.sigmoid(x)

    # Traningsalgorithmus
optimizer = optim.Adam(model.parameters(), lr=0.001)
def train(epoch):
    model.train()
    batch_id = 0
    for data, target in train_data_set:
        data = Variable(data)
        target = torch.Tensor(target)
        target = Variable(target)
        optimizer.zero_grad()
        out = model(data)
        criterion = F.binary_cross_entropy
        loss = criterion(out,target)
        loss.backward()
        optimizer.step()
        print ('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
            epoch, batch_id * len(data), len(train_data_set)*32,
                100. * batch_id / len(train_data_set), loss.item()))
        batch_id = batch_id + 1

When i change the size of the Image to 50*50 and i change the code Net like this:

class Netz(nn.Module):
def __init__(self):
    super(Netz,self).__init__()
    self.conv1 = nn.Conv2d(3, 6, kernel_size=5)
    self.conv2 = nn.Conv2d(6, 12, kernel_size=3)
    self.conv3 = nn.Conv2d(12, 18, kernel_size=3)
    self.conv4 = nn.Conv2d(18, 24, kernel_size=3)
    self.fc1 = nn.Linear(768, 1000)
    self.fc2 = nn.Linear(1000, 350)
    self.fc3 = nn.Linear(350,43)

def forward (self,x):
    x = F.relu(F.max_pool2d(self.conv1(x), 2))
    x = F.relu(F.max_pool2d(self.conv2(x), 2))
    x = F.relu(F.max_pool2d(self.conv3(x), 2))
    x = F.relu(F.max_pool2d(self.conv4(x), 2))
    x = x.view(-1,768)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return torch.sigmoid(x)

I gonne get the error:
ValueError: Target and input must have the same number of elements. target nelement (1376) != input nelement (43)

So far i see it the problem comes after the x = x.view(-1,768) it returns an Tensor with the torch.Size ([1,768]). When i use the Image Size 256*256 it returns Tensor with torch.Size ([32,4704]) and i don't get the error.

Does someone know how i can fix me problem ?

Output:
You need to be careful when using -1 in views, since it just uses the remaining size and if that doesn't correspond to your intentions, you won't immediately know that it didn't behave as expected. You should particularly avoid -1 for the batch dimension, because you can mistakenly change the batch size, which should not change and their data should be independent from each other.

Given an input of size [32, 3, 256, 256] the output after the convolutions has size [32, 24, 14, 14], which can be flattened to [32, 4704], as you anticipated in the first version. When you change the input to size [32, 3, 50, 50], the output after the convolutions has size [32, 24, 1, 1], which can clearly not be converted to size [32, 768], because flattening it, would result in a size of [32, 24]. Given that 32 * 24 = 768, you incorrectly combine the batches into one, creating a tensor of size [1, 768], and if you had used a different batch size, it wouldn't even work.

The correct input size to the first linear should be 24:

self.fc1 = nn.Linear(24, 1000)

To catch any mistake regarding the dimensions in the model, instead of later at the loss calculation, you can set the actual size without any -1 in the view, or alternatively flatten it after the batch size, either with view or torch.flatten, so an error will occur if there is a size mismatch in the linear layer:

# Reshape to [batch_size, 24]
x = x.view(x.size(0), 24)

# Flatten with view
x = x.view(x.size(0), -1)

# Flatten, starting from dimension 1 (after the batch dimension)
x = x.flatten(1)


Instruction: Dimension error in implementation of a convolutional network

Input:
I am trying to understand why my classifier has a dimension issue. Here is my code:

class convnet(nn.Module):

    def __init__(self, num_classes=1000):
        super(convnet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(kernel_size=2, stride = 2),
            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(kernel_size=2, stride = 2), #stride=2),
            nn.Conv2d(32, 64, kernel_size=3, stride=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(kernel_size=2, stride = 2),
        )

        self.classifier = nn.Sequential(
            nn.Linear(576, 128),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.Linear(128, 64),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(64),
            nn.Linear(64,num_classes),
            nn.Softmax(),
       )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x,1) #x.view(x.size(0), 256 * 6 * 6)
        x = self.classifier(x)
        return x

def neuralnet(num_classes,**kwargs):
    model = convnet(**kwargs)
    return model

So here my issue is: expected 4D input (got 2D input)

I'm quite sure that the error arises from the flatten command, however I don't really understand why as the classifier has fully dense connections. If someone knows where I'm going wrong, that would be very helpful!

Thank you

Output:
After flattening, the input to the classifier has 2 dimensions (size: [batch_size, 576]), therefore the output of the first linear layer will also have 2 dimensions (size: [batch_size, 128]). That output is then passed to nn.BatchNorm2d, which requires its input to have 4 dimensions (size: [batch_size, channels, height, width]).

If you want to use batch norm on a 2D input, you need to use nn.BatchNorm1d, which accepts either a 3D input (size: [batch_size, channels, length]) or a 2D input (size: [batch_size, length]).

self.classifier = nn.Sequential(
    nn.Linear(576, 128),
    nn.BatchNorm1d(128),
    nn.ReLU(inplace=True),
    nn.Linear(128, 64),
    nn.ReLU(inplace=True),
    nn.BatchNorm1d(64),
    nn.Linear(64,num_classes),
    nn.Softmax(),
)


Instruction: Why do I get NONE gradient of parameters in a loaded model in Pytorch, even after backword?

Input:
I have a pretrained model which was saved by

torch.save(net, 'lenet5_mnist_model')

And now I am loading it back and trying to calculate fisher information matrix like this:

precision_matrices = {}
batch_size = 32
my_model = torch.load('lenet5_mnist_model')
my_model.eval()              #  I tried to comment this off, but still no luck
for n, p in deepcopy({n: p for n, p in my_model.named_parameters()}).items()
   p = torch.tensor(p, requires_grad = True)
   p.data.zero_()
   precision_matrices[n] = variable(p.data)
for idx in range(int(images.shape[0]/batch_size)):
   x = images[idx*batch_size : (idx+1)*batch_size]
   my_model.zero_grad()
   x = Variable(x.cuda(), requires_grad = True)
   output = my_model(x).view(1,-1)
   label = output.max(1)[1].view(-1)
   loss = F.nll_loss(F.log_softmax(output, dim=1), label)
   loss = Variable(loss, requires_grad = True)
   loss.backward()
   for n, p in my_model.named_parameters():
       precision_matrices[n].data += p.grad.data**2

Finally, the above code will crash at the last line, because p.grad is NoneType. So the error is:

AttributeError: 'NoneType' object has no attribute 'data'.

Could someone provide some guidance on what caused the NoneType grad for the parameters? How should I fix this?

Output:
Your loss does not backpropagate the gradients through the model, because you are creating a new loss tensor with the value of the actual loss, which is a leaf of the computational graph, meaning that there is no history to backpropagate through.

loss.backward() needs to be called on the output of loss = F.nll_loss(F.log_softmax(output, dim=1), label).

I'm assuming that you thought you need to create a tensor with requires_grad=True, to be able to calculate the gradients. That is not the case. Tensors created with requires_grad=True are the leaves of the computational graph (they start the graph) and every operation performed on any tensor that is part of the graph is tracked such that the gradients can flow through the intermediate results to the leaves. Only tensors that need to be optimised (i.e. learnable parameters) should set requires_grad=True manually (the model's parameters do that automatically), everything else regarding the gradients is inferred. Neither x nor the loss are learnable parameters.

This confusion presumably arose due to the use of Variable. It was deprecated in PyTorch 0.4.0, which was released over 2 years ago, and all of its functionality has been merged into the tensors. Please do not use Variable.

x = images[idx*batch_size : (idx+1)*batch_size]
my_model.zero_grad()
x = x.cuda()
output = my_model(x).view(1,-1)
label = output.max(1)[1].view(-1)
loss = F.nll_loss(F.log_softmax(output, dim=1), label)
loss.backward()


Instruction: How to use list as index with pytorch

Input:
For example, a 2d tensor:

>>> t = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

And a list l = [0, 1]

If I execute t[l], then it ends up printing the 0th and first line of t.

But what if I want to use l as an index? I expect to use l to find the element at 0th row and 1th column. In other words, I expect the same result as t[0, 1] or t[0][1].

And I want to use it in more than 2d dimensions as well. Using l with length n as an index to track elements in n dimensions tensor.

Output:
IIUC You can do this for given scenario - t[tuple(l)]

t
tensor([[1, 2, 3],
        [4, 5, 6],
        [7, 8, 9]])
l
[0, 1]

t[tuple(l)]        # equivalent to t[(0,1)] or t[0,1]
tensor(2)


Instruction: Training with Pytorch: error due to CUDA memory issue

Input:
I am trying to train a model on the Cityscapes dataset, for segmentation. I use torchvision deeplabv3_resnet50 model and it's Cityscapes dataset class and transforms. In case it matters, I am running the code in Jupyter notebook.

The datasets are working, as are the dataloaders. When I attempt to train, I always get this error, at the point when the first batch is trying to be put thru the network (y_ = net(xb) in one_epoch function).

RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 6.00 GiB total capacity; 4.20 GiB already allocated; 6.87 MiB free; 4.20 GiB reserved in total by PyTorch)

What is strange, is that no matter what the batch size (bs) is, the the amount of memory free according to the error is a value a little less than the amount of memory that is trying to be allocated, e.g. for bs=16 I get:

RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 6.00 GiB total capacity; 2.90 GiB already allocated; 1.70 GiB free; 2.92 GiB reserved in total by PyTorch)

I have a much more complicated model running, that will work with bs=16. This model builds everything from scratch. But I really want to be able to use the simplicity that torchvision seems to have with it's model zoo and datasets.

My code is below, not much more than the bare essentials, enough to show if it is running ok on the GPU.

def one_epoch(net, loss, dl, opt=None, metric=None):

if opt:
    net.train()  # only affects some layers
else:
    net.eval()
    rq_stored = []
    for p in net.parameters():
        rq_stored.append(p.requires_grad)
        p.requires_grad = False

L, M = [], []
dl_it = iter(dl)
for xb, yb in tqdm(dl_it, leave=False):
    xb, yb = xb.cuda(), yb.cuda()
    y_ = net(xb)
    l = loss(y_, yb)
    if opt:
        opt.zero_grad()
        l.backward()
        opt.step()
    L.append(l.detach().cpu().numpy())
    if metric: M.append(metric(y_, yb).cpu().numpy())

if not opt:
    for p,rq in zip(net.parameters(), rq_stored): p.requires_grad = rq

return L, M

accuracy = lambda y_,yb: (y_.max(dim=1)[1] == yb).float().mean()

def fit(net, tr_dl, val_dl, loss=nn.CrossEntropyLoss(), epochs=3, lr=3e-3, wd=1e-3):

opt = optim.Adam(net.parameters(), lr=lr, weight_decay=wd)

Ltr_hist, Lval_hist = [], []
for epoch in trange(epochs):
    Ltr,  _    = one_epoch(net, loss, tr_dl,  opt)
    Lval, Aval = one_epoch(net, loss, val_dl, None, accuracy)
    Ltr_hist.append(np.mean(Ltr))
    Lval_hist.append(np.mean(Lval))
    print(f'epoch: {epoch+1}\ttraining loss: {np.mean(Ltr):0.4f}\tvalidation loss: {np.mean(Lval):0.4f}\tvalidation accuracy: {np.mean(Aval):0.2f}')

return Ltr_hist, Lval_hist

class To3ch(object):
def __call__(self, pic):
    if pic.shape[0]==1: pic = pic.repeat(3,1,1)
    return pic

bs = 1
imagenet_stats = ([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

transf = transforms.Compose([
    transforms.ToTensor(),
    To3ch(),
    transforms.Normalize(*imagenet_stats)
])

train_ds = datasets.Cityscapes('C:/cityscapes_ds', split='train', target_type='semantic', transform=transf, target_transform=transf)
val_ds = datasets.Cityscapes('C:/cityscapes_ds', split='val', target_type='semantic', transform=transf, target_transform=transf)

train_dl  = DataLoader(train_ds,  batch_size=bs,   shuffle=True,  num_workers=0)
val_dl = DataLoader(val_ds, batch_size=2*bs, shuffle=False, num_workers=0)

net = models.segmentation.deeplabv3_resnet50(num_classes=20)
fit(net.cuda(), train_dl, val_dl, loss=nn.CrossEntropyLoss(), epochs=1, lr=1e-4, wd=1e-4, plot=True)

Output:
You didn't specify, but if you're using the original Cityscapes, this OOM is completely expected.

The original Cityscapes dataset has large images (something like 1024x2048, IIRC), and it looks like you have a 6GB GPU. FYI, I cannot fit batch_size=2 in a 12GB GPU with inputs of this size.

When training DeepLab models, it is common to apply transformations on the input (e.g., random crops, resize, scaling, etc.), and it looks like you don't apply any.

When you say:

  I have a much more complicated model running, that will work with bs=16.

Perhaps you're looking at a different kind of complexity, something that has less impact on memory requirements than you think.


Instruction: Training Generator of Generative Adversarial Network (GAN) in PyTorch

Input:
I am working on implementing a Generative Adversarial Network (GAN) in PyTorch 1.5.0.

For computing the loss of the generator, I compute both the negative probabilities that the discriminator mis-classifies an all-real minibatch and an all-(generator-generated-)fake minibatch. Then, I back-propagate both parts sequentially and finally apply the step function.

Calculating and back-propagating the part of the loss which is a function of the mis-classifications of the generated fake data seems straight forward, since during back-propagation of that loss term, the backward path leads through the generator who has produced the fake data in the first place.

However, classification of all-real-data minibatches does not involve passing data through the generator. Therefore, I was wondering whether the following code snipped would still calculate gradients for the generator or whether it would not calculate any gradients at all (since the backward path does not lead through the generator and the discriminator is in eval-mode while updating the generator)?

# Update generator #
net.generator.train()
net.discriminator.eval()
net.generator.zero_grad()

# All-real minibatch
x_real = get_all_real_minibatch()
y_true = torch.full((batch_size,), label_fake).long()  # Pretend true targets were fake
y_pred = net.discriminator(x_real)  # Produces softmax probability distribution over (0=label_fake,1=label_real)

loss_real = NLLLoss(torch.log(y_pred), y_true)
loss_real.backward()
optimizer_generator.step()

If this doesn’t work as intended, how could I make it work? Thanks in advance!

Output:
No gradients are propagated to the generator, as no calculation was performed with any of the generator's parameters. The discriminator being in eval mode would not prevent the gradients from propagating to the generator, albeit they would be slightly different if you are using layers that behave differently in eval mode compared to train mode, such as dropout.

The misclassification of real images is not part of training the generator, because it doesn't gain anything from this information. Conceptually, what should the generator learn from the fact that the discriminator failed to correctly classify a real image? The sole task of the generator is to create a fake image such that the discriminator thinks it's real, therefore the only relevant information for the generator is whether the discriminator was able to identify the fake image. If the discriminator was indeed able to identify the fake image, the generator needs to adjust itself to create a more convincing fake.

Of course it's not a binary case, but the generator always tries to improve the fake image such that the discriminator is even more convinced that it was a real image. The generator's goal is not to make the discriminator be doubtful (probability of 0.5 that it's real or fake), but that the discriminator is fully convinced that it's real, even though it's fake. That's why they are adversarial, not cooperative.


Instruction: huggingface transformers bert model without classification layer

Input:
I want to do a joint-embedding from vgg16 and bert for classification.

The thing with huggingface transformers bert is that it has the classification layer which has num_labels dimension.

But, I want the output from BertPooler (768 dimensions) which I will use as a text-embedding for an extended model.

from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

This gives the following model:

BertForSequenceClassification(
...
...
        (11): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=768, out_features=2, bias=True)
)

How can I get rid of the classifier layer?

Output:
from transformers import BertModel
model = BertModel.from_pretrained('bert-base-uncased')

Output

(11): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
    )
  )
  (pooler): BertPooler(
    (dense): Linear(in_features=768, out_features=768, bias=True)
    (activation): Tanh()
  )
)

Checkout the BertModel definition here.