Instruction: Fixing incorrect dimensions in PyTorch neural network

Input:
I am trying to train my neural network, which is written in PyTorch, but I got the following traceback because of incorrect dimensions. Got the following traceback

Traceback (most recent call last):
  File "plot_parametric_pytorch.py", line 139, in <module>
    ops = opfun(X_train[smpl])
  File "plot_parametric_pytorch.py", line 92, in <lambda>
    opfun = lambda X: model.forward(Variable(torch.from_numpy(X)))
  File "/mnt_home/klee/LBSBGenGapSharpnessResearch/deepnet.py", line 77, in forward
    x = self.features(x)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/pooling.py", line 141, in forward
    self.return_indices)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/_jit_internal.py", line 209, in fn
    return if_false(*args, **kwargs)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/functional.py", line 539, in _max_pool2d
    input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: Given input size: (512x1x1). Calculated output size: (512x0x0). Output size is too small

This is all when trying to run a forward pass.
I'm pretty sure this is a small bug but I myself am new to writing PyTorch code so I am not sure if I know where it is. For reference, when I checked the dimensions of the Keras model version of this by using model.summary(), the final dimensions before flattening and adding dense layers(which I think should happen in self.classifier in pytorch, although I am not sure too) were 512 x 1 x 1.

This is my model in PyTorch:

class VGG(nn.Module):
    def __init__(self, num_classes=10):
        super(VGG, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Conv2d(64, 64, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(64, 128, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(128, 128, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(256, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(256, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Linear(512, 512, bias=False),
            nn.Dropout(0.5),
            nn.BatchNorm1d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(512, num_classes)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, 512)
        x = self.classifier(x)
        return F.log_softmax(x)

def cifar10_deep(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 10)
    return VGG(num_classes)

def cifar100_deep(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 100)
    return VGG(num_classes)

And I am trying to run the following code:

cudnn.benchmark = True
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')
X_train = np.transpose(X_train, axes=(0, 3, 1, 2))
X_test = X_test.astype('float32')
X_test = np.transpose(X_test, axes=(0, 3, 1, 2))
X_train /= 255
X_test /= 255
device = torch.device('cuda:0')

# This is where you can load any model of your choice.
# I stole PyTorch Vision's VGG network and modified it to work on CIFAR-10.
# You can take this line out and add any other network and the code
# should run just fine.
model = cifar_shallow.cifar10_shallow()
#model.to(device)

# Forward pass
opfun = lambda X: model.forward(Variable(torch.from_numpy(X)))

# Forward pass through the network given the input
predsfun = lambda op: np.argmax(op.data.numpy(), 1)

# Do the forward pass, then compute the accuracy
accfun = lambda op, y: np.mean(np.equal(predsfun(op), y.squeeze()))*100

# Initial point
x0 = deepcopy(model.state_dict())

# Number of epochs to train for
# Choose a large value since LB training needs higher values
# Changed from 150 to 30
nb_epochs = 30
batch_range = [25, 40, 50, 64, 80, 128, 256, 512, 625, 1024, 1250, 1750, 2048, 2500, 3125, 4096, 4500, 5000]

# parametric plot (i.e., don't train the network if set to True)
hotstart = False

if not hotstart:
    for batch_size in batch_range:
        optimizer = torch.optim.Adam(model.parameters())
        model.load_state_dict(x0)
        #model.to(device)
        average_loss_over_epoch = '-'
        print('Optimizing the network with batch size %d' % batch_size)
        np.random.seed(1337) #So that both networks see same sequence of batches
        for e in range(nb_epochs):
            model.eval()
            print('Epoch:', e, ' of ', nb_epochs, 'Average loss:', average_loss_over_epoch)
            average_loss_over_epoch = 0

            # Checkpoint the model every epoch
            torch.save(model.state_dict(), "./models/ShallowNetCIFAR10BatchSize" + str(batch_size) + ".pth")
            array = np.random.permutation(range(X_train.shape[0]))
            slices = X_train.shape[0] // batch_size
            beginning = 0
            end = 1

            # Training loop!
            for _ in range(slices):
                start_index = batch_size * beginning
                end_index = batch_size * end
                smpl = array[start_index:end_index]
                model.train()
                optimizer.zero_grad()
                ops = opfun(X_train[smpl]) <<----- error in this line
                tgts = Variable(torch.from_numpy(y_train[smpl]).long().squeeze())
                loss_fn = F.nll_loss(ops, tgts)
                average_loss_over_epoch += loss_fn.data.numpy() / (X_train.shape[0] // batch_size)
                loss_fn.backward()
                optimizer.step()
                beginning += 1
                end += 1

I am wondering where in my model I went wrong. I was writing the PyTorch version of the following Keras model. Any help in fixing the small bug would be appreciated!

def deepnet(nb_classes):
    global img_size
    model = Sequential()
    model.add(Conv2D(64, (3, 3), input_shape=img_size))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(Dropout(0.3))
    model.add(Conv2D(64, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

    model.add(Conv2D(128, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu')); model.add(Dropout(0.4))
    model.add(Conv2D(128, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

    model.add(Conv2D(256, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu')); model.add(Dropout(0.4))
    model.add(Conv2D(256, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu')); model.add(Dropout(0.4))
    model.add(Conv2D(256, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

    model.add(Conv2D(512, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu')); model.add(Dropout(0.4))
    model.add(Conv2D(512, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu')); model.add(Dropout(0.4))
    model.add(Conv2D(512, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

    model.add(Conv2D(512, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu')); model.add(Dropout(0.4))
    model.add(Conv2D(512, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu')); model.add(Dropout(0.4))
    model.add(Conv2D(512, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

    model.add(Flatten()); model.add(Dropout(0.5))
    model.add(Dense(512))
    model.add(BatchNormalization())
    model.add(Activation('relu')); model.add(Dropout(0.5))
    model.add(Dense(nb_classes, activation='softmax'))
    return model

Please let me know if there is an issue with the way I converted the neural network model from PyTorch to Keras. From what I understand, padding should always equal 1 in pytorch because of padding=same setting in Keras.

Output:
The first convolution doesn't use padding.

nn.Conv2d(3, 64, kernel_size=3, bias=False)

Therefore the spatial dimensions will be reduced by 2. In the case of CIFAR the input has size [batch_size, 3, 32, 32] and the output would be [batch_size, 64, 30, 30]. For all other convolutions the spatial dimensions are unchanged, but the max pooling will halve them (integer division). Since you have 5 max pooling layers, the height/width change as follows:

30 -> 15 -> 7 -> 3 -> 1 -> 0 (error)

In the Keras version you are using padding in the max pooling layers as well, which is presumably only applied if the input is not strictly divisible by 2. If you wanted to replicate that behaviour in PyTorch you would have to set the padding of the max pooling layers manually for the ones that receive an input with an odd height/width.

I don't think that using padding in max pooling with a kernel size of 2 is beneficial, especially as you are using ReLU before them, meaning that the padded max pooling just preserves the border values (it's a different story for bigger kernel sizes).

The simplest solution is to use padding in the first convolution, such that the spatial dimensions are unchanged:

nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False)

Another option would be to remove the last max pooling layer, since the height/width are already 1, but that also means that the last three convolutions are applied to only one value, since the input sizes would be [batch_size, 512, 1, 1], which kind of defeats the purpose of using a convolution.


Instruction: PyTorch: Index high dimensional tensor with two dimensional tensor

Input:
Suppose I have the following tensors:

N = 2
k = 3
d = 2

L = torch.arange(N * k * d * d).view(N, k, d, d)
L
tensor([[[[ 0,  1],
          [ 2,  3]],

         [[ 4,  5],
          [ 6,  7]],

         [[ 8,  9],
          [10, 11]]],

        [[[12, 13],
          [14, 15]],

         [[16, 17],
          [18, 19]],

         [[20, 21],
          [22, 23]]]])

index = torch.Tensor([0,1,0,0]).view(N,-1)
index
tensor([[0., 1.],
        [0., 0.]])

I now would like to use the index tensor to pick out the corresponding matrices on the second dimension, i.e. I would like to get something like:

tensor([[[[ 0,  1],
          [ 2,  3]],

         [[ 4,  5],
          [ 6,  7]]],

        [[[12, 13],
          [14, 15]],

         [[[[12, 13],
          [14, 15]]])

Any idea how I could achieve this?
Thank you so much!

Output:
Tensors can be indexed with multiple tensors specified across different dimensions (tuples of tensors), where the i-th element of each tensor are combined to create a tuple of indices, i.e. data[indices_dim0, indices_dim1] results in indexing data[indices_dim0[0], indices_dim1[0]], data[indices_dim0[1], indices_dim1[1]] and so on. They must have the same length len(indices_dim0) == len(indices_dim1).

Let's use the flat version of index (before you applied the view). Each element needs to be matched to the appropriate batch index, which would be [0, 0, 1, 1]. Also index needs to have type torch.long, because floats cannot be used as indices. torch.tensor should be preferred for creating tensors with existing data, since torch.Tensor is an alias for the default tensor type (torch.FloatTensor), whereas torch.tensor automatically uses the data type that represents the given values, but also supports the dtype argument to set the type manually, and is generally more versatile.

# Type torch.long is inferred
index = torch.tensor([0, 1, 0, 0])

# Same, but explicitly setting the type
index = torch.tensor([0, 1, 0, 0], dtype=torch.long)

batch_index = torch.tensor([0, 0, 1, 1])

L[batch_index, index]
# => tensor([[[ 0,  1],
#             [ 2,  3]],
#
#            [[ 4,  5],
#             [ 6,  7]],
#
#            [[12, 13],
#             [14, 15]],
#
#            [[12, 13],
#             [14, 15]]])

The indices are not limited to 1D tensors, but they all need to have the same size and each element is used as one index, for example with 2D tensors the indexing happens as data[indices_dim0[i][j], indices_dim1[i][j]]
With 2D tensors it happens to be much simpler to create the batch indices without having to do it manually.

index = torch.tensor([0, 1, 0, 0]).view(N, -1)
# => tensor([[0, 1],
#            [0, 0]])

# Every batch gets its index and is repeated across dim=1
batch_index = torch.arange(N).view(N, 1).expand_as(index)
# => tensor([[0, 0],
#            [1, 1]])

L[batch_index, index]


Instruction: How to create a Pytorch network with mixed categorical and continuous matrix input

Input:
I'm creating a network network that will take a matrix of continuous values along with some categorical input represented as vectors of all the classes.

Now, I'm also looking extract features from the matrix with convolution. But this would not be possible if I reduce the matrix to dimension 1 and concatenate with the class vectors.

Is there a way to concat it together as a single input? Or do I have to create two separate input layer then somehow join them after the convolution? If its the latter, what function am I looking for?

Output:
The most common approach to create continuous values from categorical data is nn.Embedding. It creates a learnable vector representation of the available classes, such that two similar classes (in a specific context) are closer to each other than two dissimilar classes.

When you have a vector of classes with size [v], the embedding would create a tensor of size [v, embedding_size], where each class is represented by a vector of length embedding_size.

num_classes = 4
embedding_size = 10

embedding = nn.Embedding(num_classes, embedding_size)

class_vector = torch.tensor([1, 0, 3, 3, 2])

embedded_classes = embedding(class_vector)
embedded_classes.size() # => torch.Size([5, 10])

How you combine them with your continuous matrix depends on your particular use case. If you just want a 1D vector you can flatten and concatenate them. On the other hand, if the matrix has meaningful dimensions that you want to keep, you should decide which dimension makes sense to concatenate on and adapt the embedding_size such that they can be concatenated.


Instruction: Pytorch transform tensor to one hot

Input:
What is the easiest way to transform tensor of shape (batch_size, height, width) filled with n values to tensor of shape (batch_size, n, height, width)?
I created solution below, but looks like there are easier and faster way to do this

def batch_tensor_to_onehot(tnsr, classes):
    tnsr = tnsr.unsqueeze(1)
    res = []
    for cls in range(classes):
        res.append((tnsr == cls).long())
    return torch.cat(res, dim=1)

Output:
You can use torch.nn.functional.one_hot.

For your case:

a = torch.nn.functional.one_hot(tnsr, num_classes=classes)
out = a.permute(0, 3, 1, 2)


Instruction: torch.rfft - fft-based convolution creating different output than spatial convolution

Input:
I implemented FFT-based convolution in Pytorch and compared the result with spatial convolution via conv2d() function. The convolution filter used is an average filter. The conv2d() function produced smoothened output due to average filtering as expected but the fft-based convolution returned a more blurry output.
I have attached the code and outputs here -

spatial convolution -

from PIL import Image, ImageOps
import torch
from matplotlib import pyplot as plt
from torchvision.transforms import ToTensor
import torch.nn.functional as F
import numpy as np

im = Image.open("/kaggle/input/tiger.jpg")
im = im.resize((256,256))
gray_im = im.convert('L')
gray_im = ToTensor()(gray_im)
gray_im = gray_im.squeeze()

fil = torch.tensor([[1/9,1/9,1/9],[1/9,1/9,1/9],[1/9,1/9,1/9]])

conv_gray_im = gray_im.unsqueeze(0).unsqueeze(0)
conv_fil = fil.unsqueeze(0).unsqueeze(0)

conv_op = F.conv2d(conv_gray_im,conv_fil)

conv_op = conv_op.squeeze()

plt.figure()
plt.imshow(conv_op, cmap='gray')

FFT-based convolution -

def fftshift(image):
    sh = image.shape
    x = np.arange(0, sh[2], 1)
    y = np.arange(0, sh[3], 1)
    xm, ym  = np.meshgrid(x,y)
    shifter = (-1)**(xm + ym)
    shifter = torch.from_numpy(shifter)
    return image*shifter

shift_im = fftshift(conv_gray_im)
padded_fil = F.pad(conv_fil, (0, gray_im.shape[0]-fil.shape[0], 0, gray_im.shape[1]-fil.shape[1]))
shift_fil = fftshift(padded_fil)
fft_shift_im = torch.rfft(shift_im, 2, onesided=False)
fft_shift_fil = torch.rfft(shift_fil, 2, onesided=False)
shift_prod = fft_shift_im*fft_shift_fil
shift_fft_conv = fftshift(torch.irfft(shift_prod, 2, onesided=False))

fft_op = shift_fft_conv.squeeze()
plt.figure('shifted fft')
plt.imshow(fft_op, cmap='gray')

original image -

spatial convolution output -

fft-based convolution output -

Could someone kindly explain the issue?

Output:
The main problem with your code is that Torch doesn't do complex numbers, the output of its FFT is a 3D array, with the 3rd dimension having two values, one for the real component and one for the imaginary. Consequently, the multiplication does not do a complex multiplication.

There currently is no complex multiplication defined in Torch (see this issue), we'll have to define our own.

A minor issue, but also important if you want to compare the two convolution operations, is the following:

The FFT takes the origin of its input in the first element (top-left pixel for an image). To avoid a shifted output, you need to generate a padded kernel where the origin of the kernel is the top-left pixel. This is quite tricky, actually...

Your current code:

fil = torch.tensor([[1/9,1/9,1/9],[1/9,1/9,1/9],[1/9,1/9,1/9]])
conv_fil = fil.unsqueeze(0).unsqueeze(0)
padded_fil = F.pad(conv_fil, (0, gray_im.shape[0]-fil.shape[0], 0, gray_im.shape[1]-fil.shape[1]))

generates a padded kernel where the origin is in pixel (1,1), rather than (0,0). It needs to be shifted by one pixel in each direction. NumPy has a function roll that is useful for this, I don't know the Torch equivalent (I'm not at all familiar with Torch). This should work:

fil = torch.tensor([[1/9,1/9,1/9],[1/9,1/9,1/9],[1/9,1/9,1/9]])
padded_fil = fil.unsqueeze(0).unsqueeze(0).numpy()
padded_fil = np.pad(padded_fil, ((0, gray_im.shape[0]-fil.shape[0]), (0, gray_im.shape[1]-fil.shape[1])))
padded_fil = np.roll(padded_fil, -1, axis=(0, 1))
padded_fil = torch.from_numpy(padded_fil)

Finally, your fftshift function, applied to the spatial-domain image, causes the frequency-domain image (the result of the FFT applied to the image) to be shifted such that the origin is in the middle of the image, rather than the top-left. This shift is useful when looking at the output of the FFT, but is pointless when computing the convolution.

Putting these things together, the convolution is now:

def complex_multiplication(t1, t2):
  real1, imag1 = t1[:,:,0], t1[:,:,1]
  real2, imag2 = t2[:,:,0], t2[:,:,1]
  return torch.stack([real1 * real2 - imag1 * imag2, real1 * imag2 + imag1 * real2], dim = -1)

fft_im = torch.rfft(gray_im, 2, onesided=False)
fft_fil = torch.rfft(padded_fil, 2, onesided=False)
fft_conv = torch.irfft(complex_multiplication(fft_im, fft_fil), 2, onesided=False)

Note that you can do one-sided FFTs to save a bit of computation time:

fft_im = torch.rfft(gray_im, 2, onesided=True)
fft_fil = torch.rfft(padded_fil, 2, onesided=True)
fft_conv = torch.irfft(complex_multiplication(fft_im, fft_fil), 2, onesided=True, signal_sizes=gray_im.shape)

Here the frequency domain is about half the size as in the full FFT, but it is only redundant parts that are left out. The result of the convolution is unchanged.


Instruction: How to initialize weights in a pytorch model

Input:
I've got a fairly straight forward problem here.
I've just finished re-configuring a network by replacing nn.Upsample with the upConv sequential container shown in the code below. I've verified that everything is lined up by running summary(UNetPP, (3, 128, 128)) which runs with no issue.

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

class blockUNetPP(nn.Module):
    def __init__(self, in_channels, middle_channels, out_channels):
        super().__init__()
        self.relu = nn.LeakyReLU(0.2, inplace=True)
        self.conv1 = nn.Conv2d(in_channels, middle_channels, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(middle_channels)
        self.conv2 = nn.Conv2d(middle_channels, out_channels, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(out_channels)

    def forward(self, x):
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        return out

class upConv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.upc = nn.Sequential(
                                 nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
                                 nn.Conv2d(in_ch, out_ch*2, 3, stride=1, padding=1),
                                 nn.BatchNorm2d(out_ch*2),
                                 nn.ReLU(inplace=True)
                                 )
    def forward(self, x):
        out = self.upc(x)
        return out

My issue is that when I try to start training the model I get the following issue:

Traceback (most recent call last):
  File "runTrain.py", line 90, in <module>
    netG.apply(weights_init)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 289, in apply
    module.apply(fn)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 290, in apply
    fn(self)
  File "D:\Thesis Models\Deep_learning_models\UNet\train\NetC.py", line 8, in weights_init
    m.weight.data.normal_(0.0, 0.02)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 594, in __getattr__
    type(self).__name__, name))
AttributeError: 'upConv' object has no attribute 'weight'

I've looked up  solutions which suggest looping over container modules, but I'm already doing this with weights_init(m). Could someone explain whats wrong with my current setup?

Output:
You are deciding how to initialise the weight by checking that the class name includes Conv with classname.find('Conv'). Your class has the name upConv, which includes Conv, therefore you try to initialise its attribute .weight, but that doesn't exist.

Either rename your class or make the condition more strict, such as classname.find('Conv2d'). The strictest approach would be to check whether it's an instance of nn.Conv2d, instead of looking at the name of the class.

def weights_init(m):
    if isinstance(m, nn.Conv2d):
        m.weight.data.normal_(0.0, 0.02)
    elif isinstance(m, nn.BatchNorm2d):
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)


Instruction: Pytorch Faster R-CNN size mismatch errors in testing

Input:
there!
When running test_net.py in pytorch1.0 Faster R-CNN and demo.py on coco dataset with faster_rcnn_1_10_9771.pth(the pretrained resnet101 model on coco dataset provided by jwyang), I encounter the same errors below :
Called with args:
Namespace(batch_size=1, cfg_file='cfgs/res101.yml', checkepoch=10, checkpoint=9771, checksession=1, class_agnostic=True, cuda=True, dataset='coco', image_dir='/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/images', load_dir='/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/models', mGPUs=True, net='res101', parallel_type=0, set_cfgs=None, vis=True, webcam_num=-1)
/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/lib/model/utils/config.py:374: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  yaml_cfg = edict(yaml.load(f))
Using config:
{'ANCHOR_RATIOS': [0.5, 1, 2],
 'ANCHOR_SCALES': [4, 8, 16, 32],
 'CROP_RESIZE_WITH_MAX_POOL': False,
 'CUDA': False,
 'DATA_DIR': '/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/data',
 'DEDUP_BOXES': 0.0625,
 'EPS': 1e-14,
 'EXP_DIR': 'res101',
 'FEAT_STRIDE': [16],
 'GPU_ID': 0,
 'MATLAB': 'matlab',
 'MAX_NUM_GT_BOXES': 20,
 'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
               'FIXED_LAYERS': 5,
               'REGU_DEPTH': False,
               'WEIGHT_DECAY': 4e-05},
 'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
 'POOLING_MODE': 'align',
 'POOLING_SIZE': 7,
 'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
 'RNG_SEED': 3,
 'ROOT_DIR': '/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0',
 'TEST': {'BBOX_REG': True,
          'HAS_RPN': True,
          'MAX_SIZE': 1000,
          'MODE': 'nms',
          'NMS': 0.3,
          'PROPOSAL_METHOD': 'gt',
          'RPN_MIN_SIZE': 16,
          'RPN_NMS_THRESH': 0.7,
          'RPN_POST_NMS_TOP_N': 300,
          'RPN_PRE_NMS_TOP_N': 6000,
          'RPN_TOP_N': 5000,
          'SCALES': [600],
          'SVM': False},
 'TRAIN': {'ASPECT_GROUPING': False,
           'BATCH_SIZE': 128,
           'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
           'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
           'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
           'BBOX_NORMALIZE_TARGETS': True,
           'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
           'BBOX_REG': True,
           'BBOX_THRESH': 0.5,
           'BG_THRESH_HI': 0.5,
           'BG_THRESH_LO': 0.0,
           'BIAS_DECAY': False,
           'BN_TRAIN': False,
           'DISPLAY': 20,
           'DOUBLE_BIAS': False,
           'FG_FRACTION': 0.25,
           'FG_THRESH': 0.5,
           'GAMMA': 0.1,
           'HAS_RPN': True,
           'IMS_PER_BATCH': 1,
           'LEARNING_RATE': 0.001,
           'MAX_SIZE': 1000,
           'MOMENTUM': 0.9,
           'PROPOSAL_METHOD': 'gt',
           'RPN_BATCHSIZE': 256,
           'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
           'RPN_CLOBBER_POSITIVES': False,
           'RPN_FG_FRACTION': 0.5,
           'RPN_MIN_SIZE': 8,
           'RPN_NEGATIVE_OVERLAP': 0.3,
           'RPN_NMS_THRESH': 0.7,
           'RPN_POSITIVE_OVERLAP': 0.7,
           'RPN_POSITIVE_WEIGHT': -1.0,
           'RPN_POST_NMS_TOP_N': 2000,
           'RPN_PRE_NMS_TOP_N': 12000,
           'SCALES': [600],
           'SNAPSHOT_ITERS': 5000,
           'SNAPSHOT_KEPT': 3,
           'SNAPSHOT_PREFIX': 'res101_faster_rcnn',
           'STEPSIZE': [30000],
           'SUMMARY_INTERVAL': 180,
           'TRIM_HEIGHT': 600,
           'TRIM_WIDTH': 600,
           'TRUNCATED': False,
           'USE_ALL_GT': True,
           'USE_FLIPPED': True,
           'USE_GT': False,
           'WEIGHT_DECAY': 0.0001},
 'USE_GPU_NMS': True}
load checkpoint /home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/models/res101/coco/faster_rcnn_1_10_9771.pth
Traceback (most recent call last):
  File "/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/demo.py", line 205, in <module>
    fasterRCNN.load_state_dict(checkpoint['model'])
  File "/home/ubuntu/users/anaconda3/envs/fasterrcnn/lib/python3.6/site-packages/torch/nn/modules/module.py", line 769, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for resnet:
    **size mismatch for RCNN_bbox_pred.weight:** copying a param with shape torch.Size([324, 2048]) from checkpoint, the shape in current model is torch.Size([4, 2048]).
    **size mismatch for RCNN_bbox_pred.bias**: copying a param with shape torch.Size([324]) from checkpoint, the shape in current model is torch.Size([4]).

Process finished with exit code 1

And here is my envs:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
blas                      1.0                         mkl    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
bzip2                     1.0.8                h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ca-certificates           2020.1.1                      0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cairo                     1.14.12              h8948797_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
certifi                   2020.4.5.1               py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cffi                      1.14.0           py36he30daa8_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
cuda100                   1.0                           0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
cycler                    0.10.0                   py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
cython                    0.29.17          py36he6710b0_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
dbus                      1.13.14              hb2f20db_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
easydict                  1.9                        py_0    https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge
expat                     2.2.6                he6710b0_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
faster-rcnn               0.1                       dev_0    <develop>
ffmpeg                    4.0                  hcdf2ecd_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
fontconfig                2.13.0               h9420a91_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
freeglut                  3.0.0                hf484d3e_5    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
freetype                  2.9.1                h8a8886c_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
glib                      2.63.1               h3eb4bd4_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
graphite2                 1.3.13               h23475e2_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
gst-plugins-base          1.14.0               hbbd80ab_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
gstreamer                 1.14.0               hb31296c_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
harfbuzz                  1.8.8                hffaf4a1_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
hdf5                      1.10.2               hba1933b_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
icu                       58.2                 he6710b0_3    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
intel-openmp              2020.1                      217    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
jasper                    2.0.14               h07fcdf6_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jpeg                      9b                   h024ee3a_2    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
kiwisolver                1.2.0            py36hfd86e86_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
ld_impl_linux-64          2.33.1               h53a641e_7    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libedit                   3.1.20181209         hc058e9b_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libffi                    3.3                  he6710b0_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libgcc-ng                 9.1.0                hdf63c60_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libgfortran-ng            7.3.0                hdf63c60_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libglu                    9.0.0                hf484d3e_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libopencv                 3.4.2                hb342d67_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libopus                   1.3.1                h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libpng                    1.6.37               hbc83047_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libprotobuf               3.11.4               hd408876_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libstdcxx-ng              9.1.0                hdf63c60_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libtiff                   4.1.0                h2733197_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libuuid                   1.0.3                h1bed415_2    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libvpx                    1.7.0                h439df22_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libxcb                    1.13                 h1bed415_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libxml2                   2.9.9                hea5a465_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
matplotlib                3.1.3                    py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
matplotlib-base           3.1.3            py36hef1b27d_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
mkl                       2020.1                      217    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
mkl-service               2.3.0            py36he904b0f_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
mkl_fft                   1.0.15           py36ha843d7b_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
mkl_random                1.1.1            py36h0573a6f_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
ncurses                   6.2                  he6710b0_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
ninja                     1.9.0            py36hfd86e86_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
numpy                     1.18.1           py36h4f9e942_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
numpy-base                1.18.1           py36hde5b4d6_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
olefile                   0.46                     py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
opencv                    3.4.2            py36h6fd60c2_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
openssl                   1.1.1g               h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pcre                      8.43                 he6710b0_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pillow                    7.1.2            py36hb39fc2d_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pip                       20.0.2                   py36_3    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pixman                    0.38.0               h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
protobuf                  3.11.4           py36he6710b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
py-opencv                 3.4.2            py36hb342d67_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pycparser                 2.20                       py_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pyparsing                 2.4.7                      py_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pyqt                      5.9.2            py36h05f1152_2    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
python                    3.6.10               h7579374_2    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
python-dateutil           2.8.1                      py_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pytorch                   1.0.0           py3.6_cuda10.0.130_cudnn7.4.1_1  [cuda100]  https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
pyyaml                    5.3.1            py36h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
qt                        5.9.7                h5867ecd_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
readline                  8.0                  h7b6447c_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
scipy                     1.2.1            py36h7c811a0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
setuptools                46.4.0                   py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
sip                       4.19.8           py36hf484d3e_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
six                       1.14.0                   py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
sqlite                    3.31.1               h62c20be_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
tensorboardx              2.0                        py_0    conda-forge
tk                        8.6.8                hbc83047_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
torchvision               0.2.0            py36h17b6947_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
tornado                   6.0.4            py36h7b6447c_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
wheel                     0.34.2                   py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
xz                        5.2.5                h7b6447c_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
yaml                      0.1.7                had09818_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zlib                      1.2.11               h7b6447c_3    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
zstd                      1.3.7                h0b5b093_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main

I tried the most useful way described here:
solution1
to __C.ANCHOR_SCALES = [4,8,16,32].

Also tried modifying the classes of pascal voc dataset to the classes of coco dataset
coco_classes = np.asarray(["__background__",
                             "person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat",
                             "traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep",
                             "cow", "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee",
                             "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard",
                             "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", "sandwich",
                             "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant", "bed",
                             "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", "microwave", "oven",
                             "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier",
                             "toothbrush"])

But still don't work for me.
Anyone help?

Output:
It says your model doesn't fit the pre-trained parameters you want to load.

Maybe check the model you're using and the .pth file and find out if they match or what.

Or post the code of your model and let's see what's going wrong.


Instruction: Is this Neural Net example I'm looking at a mistake or am I not understanding backprop?

Input:
Is this model using one relu in two places, or are gradients computed by doing a matrix multiplication of layers on both sides of one layer?

In the last layer of this simple neural net (below) during back prop it calculates the gradient for the last layer w2 by doing a matrix multiplication of y prediction - y and h_relu, which I thought was only between layers w1 and w2 not between w2 and y_pred

The line in question is near the bottom. It is grad_w2 = h_relu.t().mm(grad_y_pred).
I am confused because I thought everything was supposed to go in order forward and go in order backwards. Is this relu being used in two places?

Here is an attempt at a visual illustration of the model.

This example is from the Pytorch website. It is the second block of code on the page.

grad_w2 = h_relu.t().mm(grad_y_pred)

import torch

dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    if t % 100 == 99:
        print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())
    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

I appreciate your patience looking at this and trying to clear this up for me.
If you can try adding another layer of whieghts in the middle with another relu that might help me understand. This is what I was trying to do.

Output:
Consider the following diagram which represents the network in question. The concept of back-propagation is simply a way to quickly and intuitively apply the chain rule on a complex sequence of operations to compute the gradient of an output w.r.t. a tensor. Usually we are interested in computing the gradients of leaf tensors (tensors which are not derived from other tensors) with respect to a loss or objective. All the leaf tensors are represented as circles in the following diagram and the loss is represented by the rectangle with the L label.

Using the backward diagram we can follow the path from L to w1 and w2 in order to determine which partial derivatives we need in order to compute the gradient of L w.r.t. w1 and w2. For simplicity we will assume that all the leaf tensors are scalars so as to avoid getting into the complexities of multiplying vectors and matrices.

Using this approach the gradients of L w.r.t. w1 and w2 are

and

Something to notice is that since w2 is a leaf tensor, we only use dy/dw2 (aka grad_w2) during computation of dL/dw2 since it isn't part of the path from L to w1.


Instruction: How to use GPU while training a model?

Input:
I am running a code to train a resnet model on a kaggle notebook. I have chosen the accelerator as GPU so I haven't made any mistakes there. I am training the model using the following code:

model.cuda()
for epoch in range(10):
  model.train(True)
  trainloss=0
  for x,y in trainloader:

    x,y=x.cuda(),y.cuda()

    yhat=model(x)
    optimizer.zero_grad()
    loss=criterion(yhat,y)
    loss.backward()
    optimizer.step()
    trainloss+=loss.item()

  print('Epoch {}  Loss: {}'.format(epoch,(trainloss/len(trainloader.dataset))))
  model.eval()
  testcorrect=0
  with torch.no_grad():
    for test_x,test_y in testloader:
      test_x,test_y=test_x.cuda(),test_y.cuda()
      yhat=model(test_x)
      _,z=yhat.max(1)
      testcorrect+=(test_y==z).sum().item()

print('Model Accuracy: ',(testcorrect/len(testloader.dataset)))

Network Code:

model=torchvision.models.resnet18(pretrained=True)

num_ftrs=model.fc.in_features
model.fc=nn.Sequential(nn.Linear(num_ftrs,1000),
                        nn.ReLU(),
                        nn.Linear(1000,2)
)

If you see I have used the .cuda() function on both my model as well as the tensors(inside the training part as well as validation part). However the GPU usage shown for the kaggle notebook is 0% while my CPU usage is up to 99%. Am I missing any code which is required to train the model using the GPU?

Output:
It might be that your model doesn't give GPU enough work. Try to make your network more GPU-hungry, e.g. introduce some linear layer with a bunch of neurons, etc. to double check that in that case you see increased GPU usage. Also I noticed that the measurement is delayed by a bit, so maybe you give GPU some work which it can do in a fraction of a second and the GPU usage bar doesn't have a chance to go higher from 0%.

Maybe you could share the actual network you're using?

I can see the GPU usage going to 100% in Kaggle notebook with a toy example like this  (notice 2500 x 2500 linear layer here):

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

trainloader = [(torch.Tensor(np.random.randn(1000, 5)), torch.Tensor([1.0] * 1000))] * 1000

model = nn.Sequential(nn.Linear(5, 2500), nn.Linear(2500, 1500), nn.Linear(1500, 1))
model.cuda()
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.)
criterion = lambda x,y : ((x-y)**2).mean()

for epoch in range(10):
  for x,y in trainloader:
    x,y=x.cuda(),y.cuda()
    yhat=model(x)
    optimizer.zero_grad()
    loss=criterion(yhat,y)
    loss.backward()
    optimizer.step()
  print(epoch)


Instruction: Conv3D size doesn’t make sense with NIFTI data?

Input:
So I am writing custom dataset for medical images, with .nii (NIFTI1 format), but there is a confusion.

My dataloader returns the shape torch.Size (1,1,256,256,51) . But NIFTI volumes use anatomical axes, different coordinate system, so it doesn’t make any sense to permute the axes, which I normally would with volume made of 2D images each  stored separately in local drive with 51 slice images (or depth), as Conv3D follows the convention (N,C,D,H,W).

so torch.Size (1,1,256,256,51) (ordinarily 51 would be the depth) doesn’t follow the convention (N,C,D,H,W) , but I should not permute the axes as the data uses entirely different coordinate system ?

Output:
In pytorch 3d convolution layer naming of the 3 dimensions you do convolution on is not really important (e.g. this layer doesn't really have a special treatment for depth compared to height). All difference is coming from kernel_size argument (and also padding if you use that). If you permute the dimensions and correspondingly permute the kernel_size parameters nothing will really change. So you can either permute your input's dimensions using e.g. x.permute(0, 1, 4, 2, 3) or continue using your initial tensor with depth as the last dimension.

Just to clarify - if you wanted to use kernel_size=(2, 10, 10) on your DxHxW image, now you can instead to use kernel_size=(10, 10, 2) on your HxWxD image. If you want all your code explicitly assume that dimension order is always D, H, W then you can create tensor with permuted dimensions using x.permute(0, 1, 4, 2, 3).

Let me know if I somehow misunderstand the problem you have.


Instruction: How to correctly use CTC Loss with GRU in pytorch?

Input:
I am trying to create ASR and I am still learning so, I am just trying with a simple GRU:

MySpeechRecognition(
  (gru): GRU(128, 128, num_layers=5, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.3, inplace=False)
  (fc1): Linear(in_features=128, out_features=512, bias=True)
  (fc2): Linear(in_features=512, out_features=28, bias=True)
)

Classifies each output as one of the possible alphabets + space + blank.

Then I use CTC Loss Function and Adam optimizer:

lr = 5e-4
criterion = nn.CTCLoss(blank=28, zero_infinity=False)
optimizer = torch.optim.Adam(net.parameters(), lr=lr)

In my training loop (I am only showing the problematic area):

output, h = mynet(specs, h)
print(output.size())
output = F.log_softmax(output, dim=2)
output = output.transpose(0,1)
# calculate the loss and perform backprop
loss = criterion(output, labels, input_lengths, label_lengths)
loss.backward()

I get this error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-133-5e47e7b03a46> in <module>
     42         output = output.transpose(0,1)
     43         # calculate the loss and perform backprop
---> 44         loss = criterion(output, labels, input_lengths, label_lengths)
     45         loss.backward()
     46         # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, log_probs, targets, input_lengths, target_lengths)
   1309     def forward(self, log_probs, targets, input_lengths, target_lengths):
   1310         return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,
-> 1311                           self.zero_infinity)
   1312
   1313 # TODO: L1HingeEmbeddingCriterion

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in ctc_loss(log_probs, targets, input_lengths, target_lengths, blank, reduction, zero_infinity)
   2050     """
   2051     return torch.ctc_loss(log_probs, targets, input_lengths, target_lengths, blank, _Reduction.get_enum(reduction),
-> 2052                           zero_infinity)
   2053
   2054

RuntimeError: blank must be in label range

I am not sure why I am getting this error. I tried changing to

labels.float()

Thanks.

Output:
Your model predicts 28 classes, therefore the output of the model has size [batch_size, seq_len, 28] (or [seq_len, batch_size, 28] for the log probabilities that are given to the CTC loss). In the nn.CTCLoss you set blank=28, which means that the blank label is the class with index 28. To get the log probabilities for the blank label you would index it as output[:, :, 28], but that doesn't work, because that index is out of range, as the valid indices are 0 to 27.

The last class in your output is at index 27, hence it should be blank=27:

criterion = nn.CTCLoss(blank=27, zero_infinity=False)


Instruction: Detectron2 Panoptic FPN Model Partial Execution - TypeError: 'NoneType' object is not iterable

Input:
I am trying to extract the pre-output feature-map for the Panoptic Output of Detectron2 ResNet50 - based FPN model.

Hence, In order to get partial model outputs, I am following the official Detectron2 Modeling Documentation to Partially Execute Models.

Please find the code below:

# Setting the model Configurations
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file(
    "COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml")
)
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(
    "COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml"
)

# Build the model from config
model = build_model(cfg)

# Loading an image just for testing
im = cv2.imread("./detectron/input.jpg")
im = torch.from_numpy(im).cuda().permute(2, 0, 1).unsqueeze(0).float()

# Extracting Features - This part works fine
features = model.backbone(im)

# Extracting the proposals - Throws Error
proposals = model.proposal_generator(im, features)

Please find the error thrown by the above step shown below:

TypeError                                 Traceback (most recent call last)
<ipython-input-17-dd53471cf2d2> in <module>
----> 1 proposals = model.proposal_generator(im, features)

~/miniconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/media/Data/Documents/Python-Codes/Freelance/detectron2/detectron2/modeling/proposal_generator/rpn.py in forward(self, images, features, gt_instances)
    409
    410         if self.training:
--> 411             gt_labels, gt_boxes = self.label_and_sample_anchors(anchors, gt_instances)
    412             losses = self.losses(
    413                 anchors, pred_objectness_logits, gt_labels, pred_anchor_deltas, gt_boxes

~/miniconda3/envs/pytorch/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17

/media/Data/Documents/Python-Codes/Freelance/detectron2/detectron2/modeling/proposal_generator/rpn.py in label_and_sample_anchors(self, anchors, gt_instances)
    272         anchors = Boxes.cat(anchors)
    273
--> 274         gt_boxes = [x.gt_boxes for x in gt_instances]
    275         image_sizes = [x.image_size for x in gt_instances]
    276         del gt_instances

TypeError: 'NoneType' object is not iterable

Please let me know what am I doing wrong, and how to fix it. An explanation regarding why did this error come up would also be very helpful.

Please let me know if any other information is required.

Output:
With a bit more digging, I solved the issue. There were a couple of problems in the above code:

I did not set the model to eval mode first - model.eval(). The model needs to be set to eval fist.
The mode.proposal_generator() expects inputs in the form of ImageList object, details regarding which can be found here.

Performing the above two steps solved the issue.


Instruction: Calculating negative ELBO

Input:
I am going through the tutorial of deep Markov models where they are trying to learn the polyphonic dataset. The Link to tutorial is:

https://pyro.ai/examples/dmm.html

This model parameterises transitions and emissions with using a neural network and for variational inference part they use RNN to map observable 'x' to latent space. And in order to ensure that their model is learning something they try to maximise ELBO or minimise negative ELBO. They refer to negative ELBO as NLL. So far I understand what they are doing. However, the next step confuses me. Once they have their NLL, they divide it by sum of sequence lengths.

times = [time.time()]
for epoch in range(args.num_epochs):
    # accumulator for our estimate of the negative log likelihood
    # (or rather -elbo) for this epoch
    epoch_nll = 0.0
    # prepare mini-batch subsampling indices for this epoch
    shuffled_indices = np.arange(N_train_data)
    np.random.shuffle(shuffled_indices)

    # process each mini-batch; this is where we take gradient steps
    for which_mini_batch in range(N_mini_batches):
        epoch_nll += process_minibatch(epoch, which_mini_batch, shuffled_indices)

    # report training diagnostics
    times.append(time.time())
    epoch_time = times[-1] - times[-2]
    log("[training epoch %04d]  %.4f \t\t\t\t(dt = %.3f sec)" %
        (epoch, epoch_nll / N_train_time_slices, epoch_time))

And I don't quite understand why they are doing that. Can some explain? Are they averaging here? Insights would be appreciated.

Output:
In the tutorial, through optimisation process they are trying to reduce the loss and finally wants to compare it with reference[1] in the tutorial.

"Finally we report some diagnostic info. Note that we normalize the loss by the total number of time slices in the training set (this allows us to compare to reference [1])."

This is from the tutorial that you have provided.

Basically loss is calculated for all the mini-batches and they are normalising it such that final loss would be the loss over whole training data sequence length they have originally taken.

And when we will run the code we can the overall loss after every epoch in diagnostic report generated from logging.


Instruction: How to perform Multi output regression using RoBERTa?

Input:
I have a problem statement where I want to predict multiple continuous outputs using a text input. I tried using 'robertaforsequenceclassification' from HuggingFace library. But the documentation states that when the number of outputs in the final layer is more than 1, a cross entropy loss is used automatically as mentioned here: https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification.
But I want to use an RMSE loss in a regression setting with two classes in the final layer. How would one go about modifying it?

Output:
BertForSequenceClassification is a small wrapper that wraps the BERTModel.

It calls the models, takes the pooled output (the second member of the output tuple), and applies a classifier over it. The code is here https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1168

The simplest solution is writing your own simple wrapper class (based on the BertForSequenceClassification class) hat will do the regression that will do the regression with the loss you like.


Instruction: Why does a single Conv2d with 10x10x3 take up 850mb of gpu

Input:
In Pytorch I am optimizing a model. If I run the following code, nvidia-smi shows that I am using  850MiB /  7979MiB of memory on my gpu. Why would this be the case?

  with torch.no_grad():
      A = nn.Conv2d(10,10,3).cuda()

I imagine there are some overhead or a default allocation size specified somewhere but I could not find such documentation. I do recall that tensorflow had a setting to limit the amount of memory allocated.

Related Git Issue

Output:
The convolution does not occupy that much memory. You can verify this with torch.cuda.memory_allocated, which shows the memory that is occupied by all tensors in bytes:

torch.cuda.memory_allocated() # => 0

A = nn.Conv2d(10,10,3).cuda()

torch.cuda.memory_allocated() # => 4608

The convolution only uses 4608 bytes.

nvidia-smi shows higher memory usage for two separate reasons.

Caching Memory Allocator

PyTorch uses a caching memory allocator, meaning that it holds onto more memory than necessary to avoid device synchronisations.

From PyTorch CUDA Semantics - Memory Management:

  PyTorch uses a caching memory allocator to speed up memory allocations. This allows fast memory deallocation without device synchronizations. However, the unused memory managed by the allocator will still show as if used in nvidia-smi. You can use memory_allocated() and max_memory_allocated() to monitor memory occupied by tensors, and use memory_reserved() and max_memory_reserved() to monitor the total amount of memory managed by the caching allocator.

CUDA Context

When CUDA is first initialised, it creates a context that manages the control of the device. Most notably, the context contains the code of all the different CUDA kernels, of which PyTorch has a large number. The size of the context also varies across different GPU architectures. Some details are discussed in Issue #20532 - Couple hundred MB are taken just by initializing cuda .

The memory you are observing is almost exclusively attributed to the CUDA context.


Instruction: Shape of target and predictions tensors in PyTorch loss functions

Input:
I am confused with the input shapes for tensors in nn.CrossEntropyLoss.
I am trying to implement a simple autoencoder for text sequences. The core of my problem can be illustrated by the following code

predictions = torch.rand(2, 3, 4)
target = torch.rand(2, 3)
print(predictions.shape)
print(target.shape)
nn.CrossEntropyLoss(predictions.transpose(1, 2), target)

In my case predictions has the shape (time_step, batch_size, vocabulary_size) while target has the shape (time_step, batch_size). Next I am transposing the predictions as per description which says that the second dimension of predictions should be the number of classes - vocabulary_size in my case. The code returns an error RuntimeError: bool value of Tensor with more than one value is ambiguous. Could someone please enlighten me how to use the damn thing? Thank you in advance!

Output:
You are not calling the loss function, but you are building it. The signature of the nn.CrossEntropyLoss constructor is:

nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean')

You are setting the predictions as the weight and the target as size_average,
where weight is an optional rescaling of the classes and size_average is deprecated, but expects a boolean. The target is a tensor of size [2, 3], which cannot be converted to a boolean.

You need to create the loss function first, as you don't use any of the optional parameters of the constructor, you don't specify any of them.

# Create the loss function
cross_entropy = nn.CrossEntropyLoss()

# Call it to calculate the loss for your data
loss = cross_entropy(predictions.transpose(1, 2), target)

Alternatively, you can directly use the functional version nn.functional.cross_entropy:

import torch.nn.functional as F

loss = F.cross_entropy(predictions.transpose(1, 2), target)

The advantage of the class version, compared to the functional version, is that you only need to specify the extra parameters once (such as the weight) instead of having to supply them manually each time.

Regarding the dimensions of the tensors, the batch size must be the first dimension, because the losses are averaged per element in the batch, so you have tensor of losses with size [batch_size]. If you used reduction="none", you would get back theses losses per element in the batch, but by default (reduction="mean") the mean of these losses is returned. That result would be different if the mean is taken across time steps rather than batches.

Lastly, the targets need to be the class indices, which means they need to have type torch.long not torch.float. In this randomly chosen example, you could create the random classes with torch.randint.

predictions = torch.rand(2, 3, 4)
target = torch.randint(4, (2, 3))

# Reorder the dimensions
# From: [time_step, batch_size, vocabulary_size]
# To: [batch_size, vocabulary_size, time_step]
predictions = predictions.permute(1, 2, 0)
# From: [time_step, batch_size]
# To: [batch_size, time_step]
target = target.transpose(0, 1)

F.cross_entropy(predictions, target)


Instruction: What are saved in optimizer's state_dict? what "state","param_groups" stands for?

Input:
When we use Adam optimizer, if we want to continue train a network from a pretrained model, we not only should load "model.state_dict", but also "optimizer.state_dict". And, if we modified our network's structure, we should also modify saved optimizer's state_dict to make our loading successful.

But I don't understand some params in saved "optimizer.state_dict". like optim_dict["state"] (dict_keys(['step', 'exp_avg', 'exp_avg_sq', 'max_exp_avg_sq']))and optim_dict['param_groups'][0]['params']. There are many of numbers like these:

 b['optimizer_state_dict']['state'].keys()
Out[71]: dict_keys([140623218628000, 140623218628072, 140623218628216, 140623218628360, 140623218628720, 140623218628792, 140623218628936, 140623218629080, 140623218629656, 140623218629728, 140623218629872, 140623218630016, 140623218630376, 140623218630448, 140623218716744, 140623218716816, 140623218717392, 140623218717464, 140623218717608, 140623218717752, 140623218718112, 140623218718184, 140623218718328, 140623218718472, 140623218719048, 140623218719120, 140623218719264, 140623218719408, 140623218719768, 140623218719840, 140623218719984, 140623218720128, 140623218720704, 140623209943112, 140623209943256, 140623209943400, 140623209943760, 140623209943832, 140623209943976, 140623209944120, 140623209944696, 140623209944768, 140623209944912, 140623209945056, 140623209945416, 140623209945488, 140623209945632, 140623209945776, 140623209946352, 140623209946424, 140623209946568, 140623209946712, 140623209947072, 140623210041416, 140623210041560, 140623210041704, 140623244033768, 140623244033840, 140623244033696, 140623244033912, 140623244033984, 140623244070984, 140623244071056, 140623244071128, 140623429501576, 140623244071200, 140623244071272, 140623244071344, 140623244071416, 140623244071488, 140623244071560, 140623244071632, 140623244071848, 140623244071920, 140623244072064, 140623244072208, 140623244072424, 140623244072496, 140623244072640, 140623244072784, 140623244073216, 140623244073288, 140623244073432, 140623244073576, 140623244073792, 140623244073864, 140623244074008, 140623244074152, 140623244074584, 140623244074656, 140623244074800, 140623244074944, 140623218540760, 140623218540832, 140623218540976, 140623218541120, 140623218541552, 140623218541624, 140623218541768, 140623218541912, 140623218542128, 140623218542200, 140623218542344, 140623218542488, 140623218542920, 140623218542992, 140623218543136, 140623218543280, 140623218543496, 140623218543568, 140623218543712, 140623218543856, 140623218544288, 140623218544360, 140623218544504, 140623218626632, 140623218626992, 140623218627064, 140623218627208, 140623218627352, 140623218627784, 140623218629440, 140623218717176, 140623218718832, 140623218720488, 140623209944480, 140623209946136, 140623210043000])

In [44]: b['optimizer_state_dict']['state'][140623218628072].keys()
Out[44]: dict_keys(['step', 'exp_avg', 'exp_avg_sq', 'max_exp_avg_sq'])

In [45]: b['optimizer_state_dict']['state'][140623218628072]['exp_avg'].shape
Out[45]: torch.Size([480])

Output:
In contrast to model's state_dict, which saves learnable parameters, the optimizer's state_dict contains information about the optimizer’s state (parameters to be optimized), as well as the hyperparameters used.

All optimizers in PyTorch need to inherit from the base class torch.optim.Optimizer. It requires two entries:

params (iterable) – an iterable of torch.Tensors or dicts. Specifies what Tensors should be optimized.
defaults (dict): a dict containing default values of optimization options (used when a parameter group doesn’t specify them).

In addition to that, optimizers also support specifying per-parameter options.

  To do this, instead of passing an iterable of Tensors, pass in an iterable of dicts. Each of them will define a separate parameter group, and should contain a params key, containing a list of parameters belonging to it.

Consider an example,

optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)

Here, we have provided the a) params, b) default hyperparameters: lr, momentum, and c) a parameter group. In this case, the model.base’s parameters will use the default learning rate of 1e-2, model.classifier’s parameters will use a learning rate of 1e-3, and a momentum of 0.9 will be used for all parameters.

The step (optimizer.step()) performs a single optimization step (parameter update), which changes the state of the optimizer.

Now, coming to optimizer's state_dict, it returns the state of the optimizer as a dict. It contains two entries:

state - a dict holding current optimization state.
param_groups - a dict containing all parameter groups (as discussed above)

Some of the hyperparameters are specific to the optimizer or model used e.g.  (used in Adam)

exp_avg: exponential moving average of gradient values
exp_avg_sq: exponential moving average of squared gradient values


Instruction: What happens when we call cpu().data.numpy() on a PyTorch tensor?

Input:
I am working on a project and need to pass the data in loss tensor to a plotting library.

What happens when I perform this call -> loss.cpu().data.numpy()

Is there a risk of detaching the tensor from the computation graph?

Output:
.cpu() copies the tensor to the CPU, but if it is already on the CPU nothing changes.

.numpy() creates a NumPy array from the tensor. The tensor and the array share the underlying memory, therefore if the NumPy array is modified in-place, the changes will be reflected in the original tensor. If you plan to make in-place modifications to the NumPy array, you should generally create a copy of it. In the case where loss was on the GPU, loss.cpu() already creates a copy, hence the in-place modifications would only affect the intermediate CPU tensor, which you aren't using.

  Is there a risk of detaching the tensor from the computation graph?

No, the original tensor loss is not affected by this in regards to the computational graph.


Instruction: resnet50.to() function on a non-NVIDIA GPU

Input:
I am trying to convert the pretrained ResNet50 model to be used on GPU using the Pytorch function resnet50.to().

The problem is that I am using an Intel Iris Plus Graphics 655 1536 MB GPU on Mac, and I don't know what argument to pass to the function as I only found the one for NVIDIA GPUs (resnet50.to('cuda:0')).

Output:
PyTorch uses Nvidia's CUDA API for all GPU interactions. Other GPUs that don't utilise the CUDA API (such as AMD or Intel GPUs) are therefore not supported.

If you don't have an Nvidia GPU, you cannot run PyTorch on the GPU.


Instruction: Replacing nn.Upsample with alternative upsample operation

Input:
I have a UNet++(view in private, code for model at the bottom of the article) which I'm trying to reconfigure. I'm getting some artifacts in some images so I'm following this article which suggests doing upsampling then a convolution operation.

I'm replacing the up-sample layers with sequential operation shown below but my model isn't learning. I suspect its to do with how I've configured the channels so I'd like another opinion.

Old up-sample operation:

self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)

New operations:

class upConv(nn.Module):
"""
Up sampling/ deconv block by factor of 2
"""
def __init__(self, in_ch, out_ch):
    super().__init__()
    self.upc = nn.Sequential(
                             nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
                             nn.Conv2d(in_ch, out_ch*2, 3, stride=1, padding=1),
                             nn.BatchNorm2d(out_ch*2),
                             nn.ReLU(inplace=True)
                             )
def forward(self, x):
    out = self.upc(x)
    return out

My question is do these two operations have the same output/function within my model?

Output:
  Do these two operations have the same output/function within my model?

If out_ch*2 == in_ch, then: yes, they have the same output shape.

If the input x is the output of a BatchNorm+ReLU op, then they could be even more similar.


Instruction: Measuring F1 score for multiclass classification natively in PyTorch

Input:
I am trying to implement the macro F1 score (F-measure) natively in PyTorch instead of using the already-widely-used sklearn.metrics.f1_score in order to calculate the measure directly on the GPU.

From what I understand, in order to compute the macro F1 score, I need to compute the F1 score with the sensitivity and precision for all labels, then take the average of all these.

My attempt

My current implementation looks like this:

def confusion_matrix(y_pred: torch.Tensor, y_true: torch.Tensor, n_classes: int):
    conf_matrix = torch.zeros([n_classes, n_classes], dtype=torch.int)
    y_pred = torch.argmax(y_pred, 1)
    for t, p in zip(y_true.view(-1), y_pred.view(-1)):
        conf_matrix[t.long(), p.long()] += 1
    return conf_matrix

def forward(self, y_pred: torch.Tensor, y_true: torch.Tensor) -> torch.Tensor:
    conf_matrix = confusion_matrix(y_pred, y_true, self.classes)
    TP = conf_matrix.diag()
    f1_scores = torch.zeros(self.classes, dtype=torch.float)
    for c in range(self.classes):
        idx = torch.ones(self.classes, dtype=torch.long)
        idx[c] = 0
        FP = conf_matrix[c, idx].sum()
        FN = conf_matrix[idx, c].sum()
        sensitivity = TP[c] / (TP[c] + FN + self.epsilon)
        precision = TP[c] / (TP[c] + FP + self.epsilon)
        f1_scores[c] += 2.0 * ((precision * sensitivity) / (precision + sensitivity + self.epsilon))
    return f1_scores.mean()

self.classes is the number of labels and self.epsilon is a very small value set to 10-e12 which prevents DivisionByZeroError.

When training, I compute the measure for every batch and take the average of all measures as the final score.

Problem

The problem is that when I compare my custom F1 score with sklearn's macro F1 score, they are rarely equal.

# example 1
eval_cce 0.5203, eval_f1 0.8068, eval_acc 81.5455, eval_f1_sci 0.8023,
test_cce 0.4784, test_f1 0.7975, test_acc 82.6732, test_f1_sci 0.8097
# example 2
eval_cce 0.3304, eval_f1 0.8211, eval_acc 87.4955, eval_f1_sci 0.8626,
test_cce 0.3734, test_f1 0.8183, test_acc 85.4996, test_f1_sci 0.8424
# example 3
eval_cce 0.4792, eval_f1 0.7982, eval_acc 81.8482, eval_f1_sci 0.8001,
test_cce 0.4722, test_f1 0.7905, test_acc 82.6533, test_f1_sci 0.8139

While I have tried to scan the internet, most cases cover binary classification. I have yet been able to discover an example to attempts to do what I am trying to.

My Question

Is there any obvious issue with my attempt?

Update (10.06.2020)

I have yet to figure out my mistake. Due to time constraint, I decided to just use the F1 macro score provided by sklearn. While it cannot work directly with GPU tensors, it is fast enough for my case anyway.

However, it would be awesome if anybody can figure this out, so that anybody else that might stumble upon this issue can get their problem resolved.

Output:
I have written my own implementation in Pytorch some time ago:
from typing import Tuple

import torch

class F1Score:
    """
    Class for f1 calculation in Pytorch.
    """

    def __init__(self, average: str = 'weighted'):
        """
        Init.

        Args:
            average: averaging method
        """
        self.average = average
        if average not in [None, 'micro', 'macro', 'weighted']:
            raise ValueError('Wrong value of average parameter')

    @staticmethod
    def calc_f1_micro(predictions: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
        """
        Calculate f1 micro.

        Args:
            predictions: tensor with predictions
            labels: tensor with original labels

        Returns:
            f1 score
        """
        true_positive = torch.eq(labels, predictions).sum().float()
        f1_score = torch.div(true_positive, len(labels))
        return f1_score

    @staticmethod
    def calc_f1_count_for_label(predictions: torch.Tensor,
                                labels: torch.Tensor, label_id: int) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Calculate f1 and true count for the label

        Args:
            predictions: tensor with predictions
            labels: tensor with original labels
            label_id: id of current label

        Returns:
            f1 score and true count for label
        """
        # label count
        true_count = torch.eq(labels, label_id).sum()

        # true positives: labels equal to prediction and to label_id
        true_positive = torch.logical_and(torch.eq(labels, predictions),
                                          torch.eq(labels, label_id)).sum().float()
        # precision for label
        precision = torch.div(true_positive, torch.eq(predictions, label_id).sum().float())
        # replace nan values with 0
        precision = torch.where(torch.isnan(precision),
                                torch.zeros_like(precision).type_as(true_positive),
                                precision)

        # recall for label
        recall = torch.div(true_positive, true_count)
        # f1
        f1 = 2 * precision * recall / (precision + recall)
        # replace nan values with 0
        f1 = torch.where(torch.isnan(f1), torch.zeros_like(f1).type_as(true_positive), f1)
        return f1, true_count

    def __call__(self, predictions: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
        """
        Calculate f1 score based on averaging method defined in init.

        Args:
            predictions: tensor with predictions
            labels: tensor with original labels

        Returns:
            f1 score
        """

        # simpler calculation for micro
        if self.average == 'micro':
            return self.calc_f1_micro(predictions, labels)

        f1_score = 0
        for label_id in range(1, len(labels.unique()) + 1):
            f1, true_count = self.calc_f1_count_for_label(predictions, labels, label_id)

            if self.average == 'weighted':
                f1_score += f1 * true_count
            elif self.average == 'macro':
                f1_score += f1

        if self.average == 'weighted':
            f1_score = torch.div(f1_score, len(labels))
        elif self.average == 'macro':
            f1_score = torch.div(f1_score, len(labels.unique()))

        return f1_score

You can test it in the following way:
from sklearn.metrics import f1_score
import numpy as np
errors = 0
for _ in range(10):
    labels = torch.randint(1, 10, (4096, 100)).flatten()
    predictions = torch.randint(1, 10, (4096, 100)).flatten()
    labels1 = labels.numpy()
    predictions1 = predictions.numpy()

    for av in ['micro', 'macro', 'weighted']:
        f1_metric = F1Score(av)
        my_pred = f1_metric(predictions, labels)

        f1_pred = f1_score(labels1, predictions1, average=av)

        if not np.isclose(my_pred.item(), f1_pred.item()):
            print('!' * 50)
            print(f1_pred, my_pred, av)
            errors += 1

if errors == 0:
    print('No errors!')


Instruction: Am I understanding PyTorch's add_ and mul_ correctly?

Input:
In this notebook the author writes the following nesterov update:

def nesterov_update(w, dw, v, lr, weight_decay, momentum):
    dw.add_(weight_decay, w).mul_(-lr)
    v.mul_(momentum).add_(dw)
    w.add_(dw.add_(momentum, v))

As I understand it, a.add(b) in PyTorch implements a+b and a.add(b,c) implements a+(b*c), because b is in the slot of the alpha parameter. And lastly, add_ does the in-place version of add.

Q: Am I right so far?

Then, if I were to sketch the above nesterov update in an expanded form that illustrates the logic, I would write:

dw = -lr*(dw + weight_decay*w)
v = v*momentum + dw
w = w + dw + momentum*v

Q: is this correct?

I'm not planning to use the above expanded "code," I'm just writing it this way to try communicate what I'm understanding that it's doing, to check.

Output:
It is important to note the PyTorch version (1.1.0) the tutorial is using. According to 1.1.0, function prototype for torch.add is torch.add(input, value=1, other, out=None). So, your interpretation of the following line:

dw.add_(weight_decay, w)

as: dw = dw + weight_decay * w is correct. So, the answer to your first question is, yes, you are right.

However, with the latest versions of PyTorch, you would get an error if torch.add is used in the same fashion.

a = torch.FloatTensor([0, 1.0, 2.0, 3.0])
b = torch.FloatTensor([0, 4.0, 5.0, 6.0])
c = 1.0
z = a.add(b, c)

The above code gives: (In PyTorch 1.5.0)

TypeError: add() takes 1 positional argument but 2 were given

However, if you perform the following, then it works fine.

z = a.add(b, alpha=c)

Note that, the prototype of torch.add is now: torch.add(input, other, *, alpha=1, out=None)

The answer to your second question is, yes, you are right.


Instruction: In PyTorch, what's the difference between training an RNN to predict the last word given a sequence, vs predicting the entire sequence shifted?

Input:
Let's say I'm trying to train an RNN language model in PyTorch. Suppose I iterate over batches of word sequences, and that each training batch tensor has the following shape:

data.shape = [batch_size, sequence_length, vocab_dim]

My question is, what's the difference between using only the last word in each sequence as the target label:

X = data[:,:-1]
y = data[:,-1]

and training to minimize loss using a softmax prediction of the last word,

vs setting the target to be the entire sequence shifted right:

X = data[:,:-1]
y = data[:,1:]

and training to minimize the sum of losses of each predicted word in the shifted sequence?

What's the correct approach here? I feel like i've seen both examples online. Does this also have to do with loop unrolling vs BPTT?

Output:
Consider the sequence prediction problem a b c d
where you want to train an RNN via teacher forcing.

If you only use the last word in the sentence, you are doing the following classification problem (on the left is the input; on the right is the output you're supposed to predict):

a b c -> d

For your second approach, where y is set to be the entire sequence shifted right, you are doing three classification problems:

a -> b
a b -> c
a b c -> d

The task of predicting the intermediate words in a sequence is crucial for training a useful RNN (otherwise, you would know how to get from c given a b, but you wouldn't know how to proceed after just a).

An equivalent thing to do would be to do define your training data as both the complete sequence a b c d and all incomplete sequences (a b, a b c). Then if you were to do just the "last word" prediction as mentioned previously, you would end up with the same supervision as the formulation where y is the entire sequence shifted right. But this is computationally wasteful - you don't want to rerun the RNN on both a b and a b c (the state you get from a b can be reused to obtain the state after consuming a b c).

In other words, the point of doing the "shift y right" is to split a single sequence (a b c d) of length N into N - 1 independent classification problems of the form: "given words up to time t, predict word t + 1", while needing just one RNN forward pass.


Instruction: Pytorch dataloader from csv of file paths and labels

Input:
I have a csv file for train and test datasets that contains the file location and the label. The head of this data frame is:

df.head()
Out[46]:
             file_path  label
0  \\images\\29771.png      0
1  \\images\\55201.png      0
2  \\images\\00715.png      1
3  \\images\\33214.png      0
4  \\images\\99841.png      1

I have multiple locations for the file paths, and limited space, so I can't copy them into \0 and \1 folder locations. How can I use this data frame to create a pytorch dataloader and/or dataset object?

Output:
Just write a custom __getitem__ method for your dataset.

class MyData(Dataset):
    def __init__(self, df):
        self.df = df

    def __len__(self):
        return self.df.shape[0]

    def __getitem__(self, index):
        image = load_image(self.df.file_path[index])
        label = self.df.label[index]

        return image, label

Where load_image is a function that reads the filename into whatever format you need.


Instruction: I modified a few layers to an example of a neural network just to see if I could. What's wrong with it?

Input:
A simple neural network I found had the layers w1, Relu, and w2. I tried to add a new weight layer in the middle and a second Relu after it. So, the layers are as follows w1, Relu, w_mid, Relu, and w2.
It is much much slower than the original 3 layer network if it works at all. I'm not sure if everything is getting a forward pass and if back prop is working across every part it is supposed to.
The neural network is from this link. It is the third block of code down the page.

This is the code I changed.
Below it is the original.

    import torch
    dtype = torch.float
    device = torch.device("cpu")
    #device = torch.device("cuda:0") # Uncomment this to run on GPU

    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    N, D_in, H, D_out = 64, 250, 250, 10

    # Create random input and output data
    x = torch.randn(N, D_in, device=device, dtype=dtype)
    y = torch.randn(N, D_out, device=device, dtype=dtype)

    # Randomly initialize weights
    w1 = torch.randn(D_in, H, device=device, dtype=dtype)
    w_mid = torch.randn(H, H, device=device, dtype=dtype)
    w2 = torch.randn(H, D_out, device=device, dtype=dtype)

    learning_rate = 1e-5
    for t in range(5000):
        # Forward pass: compute predicted y
        h = x.mm(w1)
        h_relu = h.clamp(min=0)
        k = h_relu.mm(w_mid)
        k_relu = k.clamp(min=0)
        y_pred = k_relu.mm(w2)

        # Compute and print loss
        loss = (y_pred - y).pow(2).sum().item()
        if t % 1000 == 0:
            print(t, loss)

        # Backprop to compute gradients of w1, mid, and w2 with respect to loss
        grad_y_pred = (y_pred - y) * 2
        grad_w2 = k_relu.t().mm(grad_y_pred)
        grad_k_relu = grad_y_pred.mm(w2.t())
        grad_k = grad_k_relu.clone()
        grad_k[k < 0] = 0
        grad_mid = h_relu.t().mm(grad_k)
        grad_h_relu = grad_k.mm(w1.t())
        grad_h = grad_h_relu.clone()
        grad_h[h < 0] = 0
        grad_w1 = x.t().mm(grad_h)

        # Update weights
        w1 -= learning_rate * grad_w1
        w_mid -= learning_rate * grad_mid
        w2 -= learning_rate * grad_w2

The loss is ..
0 1904074240.0
1000 639.4848022460938
2000 639.4848022460938
3000 639.4848022460938
4000 639.4848022460938

This is the original code from the Pytorch website.

    import torch

    dtype = torch.float
    #device = torch.device("cpu")
    device = torch.device("cuda:0") # Uncomment this to run on GPU

    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    N, D_in, H, D_out = 64, 1000, 100, 10

    # Create random input and output data
    x = torch.randn(N, D_in, device=device, dtype=dtype)
    y = torch.randn(N, D_out, device=device, dtype=dtype)

    # Randomly initialize weights
    w1 = torch.randn(D_in, H, device=device, dtype=dtype)
    w2 = torch.randn(H, D_out, device=device, dtype=dtype)

    learning_rate = 1e-6
    for t in range(500):
        # Forward pass: compute predicted y
        h = x.mm(w1)
        h_relu = h.clamp(min=0)
        y_pred = h_relu.mm(w2)

        # Compute and print loss
        loss = (y_pred - y).pow(2).sum().item()
        if t % 100 == 99:
            print(t, loss)

        # Backprop to compute gradients of w1 and w2 with respect to loss
        grad_y_pred = 2.0 * (y_pred - y)
        grad_w2 = h_relu.t().mm(grad_y_pred)
        grad_h_relu = grad_y_pred.mm(w2.t())
        grad_h = grad_h_relu.clone()
        grad_h[h < 0] = 0
        grad_w1 = x.t().mm(grad_h)

        # Update weights using gradient descent
        w1 -= learning_rate * grad_w1
        w2 -= learning_rate * grad_w2

Output:
The calculations for the gradient of h_relu is not correct.

grad_h_relu = grad_k.mm(w1.t())

That should be a w_mid not w1:

grad_h_relu = grad_k.mm(w_mid.t())

Other than that, the calculations are correct, but you should lower the learning rate, as the gradients are very big at the beginning, making the weights very large and that leads to overflowing values (infinity), which in turn produce NaN losses and gradients. This is known as exploding gradients.

In your example a learning rate of 1e-8 seems to work.


Instruction: Differences in encoder - decoder models between Keras and Pytorch

Input:
There seem to be significant, fundamental differences in construction of encoder-decoder models between keras and pytorch. Here is keras' enc-dec blog and here is pytorch's enc-dec blog.

Some differences I noticed are the following:

Keras' model directly feeds input to LSTM layer. Whereas Pytorch uses an embedding layer for both the encoder and decoder.
Pytorch uses an embedding layer with no activation in the encoder but uses relu activation for the embedding layer in the decoder.

Given these observations, my questions are the following:

My understanding is the following, is it correct? The embedding layer is not strictly required but it helps in finding a better and denser representation of the input. It is optional and you can still build a good model without the embedding layer (dependent on the problem). This is why Keras chose not to use it in this particular example. Is this a sound reason or is there more to the story?
Why use an activation for the embedding layer in the decoder but not the encoder?
Why use 'relu' as the activation instead of 'tanh', etc for the embedding layer? What's the intuition here? I've only seen 'relu' applied to data that has spatial relation, not temporal relation.

Output:
You have a wrong understanding of encoder-decoder models. First of all, please note Keras and Pytorch are two deep learning frameworks, while encoder-decoder is a type of neural network architecture. So, you need to understand how encoder-decoder works in the first place and then revise their architecture as per your need. Now, let me come back to your questions.

Embedding layer converts one-hot encoding representations into low-dimensional vector representations. For example, we have a sentence I love programming. We want to translate this sentence into German using an encoder-decoder network. So, the first step is to first convert the words in the input sentence into a sequence of vector representations, and this can be done using an embedding layer. Please note, the use of Keras or Pytorch doesn't matter. You can think, how would you give a natural language sentence as input to an LSTM? Obviously, you first need to convert them into vectors.
There is no such rule that you should use an activation layer in the embedding layer for the decoder, but not in the encoder. Remember, activation functions are non-linear functions. So, applying a non-linearity has different consequences but it has nothing to do with the encoder-decoder framework.
Again, the choice of activation function depends on other factors, not on encoder or decoder or a specific type of neural network architecture. I suggest you read the characteristics of the popular activation functions that are used in neural networks. Also, do not come into conclusions after observing a few use cases. Such conclusions are dangerous.


Instruction: Convert dictionary to tensors, and back

Input:
I started with a dictionary object:

{"train": [{"input": [[3, 1, 2], [3, 1, 2], [3, 1, 2]], "output": [[4, 5, 6], [4, 5, 6], [4, 5, 6]]}, {"input": [[2, 3, 8], [2, 3, 8], [2, 3, 8]], "output": [[6, 4, 9], [6, 4, 9], [6, 4, 9]]}]}

I then want to add padding to each input and output object in a loop. So I converted each input and output to a tensor so I could then use  F.pad to add padding. Result of the first input:

tensor([[0, 0, 0, 0, 0],
        [0, 3, 1, 2, 0],
        [0, 3, 1, 2, 0],
        [0, 3, 1, 2, 0],
        [0, 0, 0, 0, 0]]).

result of the first output:

tensor([[0, 0, 0, 0, 0],
        [0, 4, 5, 6, 0],
        [0, 4, 5, 6, 0],
        [0, 4, 5, 6, 0],
        [0, 0, 0, 0, 0]])

So that works fine.  Now, I want to reconstruct the generated tensors into the same form as the original dictionary, so that it will look like this:

{"train": [{"input": [[0, 0, 0, 0, 0], [0, 3, 1, 2, 0], [0, 3, 1, 2, 0], [0, 3, 1, 2, 0], [0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 0], [0, 4, 5, 6, 0], [0, 4, 5, 6, 0], [0, 4, 5, 6, 0], [0, 0, 0, 0, 0]]}, {"input": [[0, 0, 0, 0, 0],
        [0, 2, 3, 8, 0],
        [0, 2, 3, 8, 0],
        [0, 2, 3, 8, 0],
        [0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 0],
        [0, 6, 4, 9, 0],
        [0, 6, 4, 9, 0],
        [0, 6, 4, 9, 0],
        [0, 0, 0, 0, 0]]}]}

I can see a string concatenation way that might work:

composedString = '{"train": [{"input": ' + tensor1 + tensor2

or something like that. But given that there are different numbers of elements in the various arrays, it seems like a loop nightmare. I'm thinking there's got to be a better way. Anyone know what it is?

Thanks.

Output:
Does the following serve your purpose?

in_dict = {"train": [{"input": [[3, 1, 2], [3, 1, 2], [3, 1, 2]], "output": [[4, 5, 6], [4, 5, 6], [4, 5, 6]]}, {"input": [[2, 3, 8], [2, 3, 8], [2, 3, 8]], "output": [[6, 4, 9], [6, 4, 9], [6, 4, 9]]}]}

train_examples = []
for item in in_dict['train']:
    in_tensor = torch.Tensor(item['input'])
    out_tensor = torch.Tensor(item['output'])
    train_examples.append([in_tensor, out_tensor])

out_dict = {'train': []}
for item in train_examples:
    out_dict['train'].append({
        'input': item[0].tolist(),
        'output': item[1].tolist()
    })

print(out_dict)


Instruction: Does deleting intermediate tensors affect the computation graph in PyTorch?

Input:
To free up memory, I was wondering if it was possible to remove intermediate tensors in the forward method of my model. Here's a minimalized example scenario:

def forward(self, input):
    x1, x2 = input
    x1 = some_layers(x1)
    x2 = some_layers(x2)
    x_conc = torch.cat((x1,x2),dim=1)
    x_conc = some_layers(x_conc)
    return x_conc

Basically, the model passes two tensors through two separate blocks, and then concatenates the results. Further operations are applied on that concatenated tensor. Will it affect the computation graph if I run del x1 and del x2 after creating x_conc?

Output:
PyTorch will store the x1, x2 tensors in the computation graph if you want to perform automatic differentiation later on. Also, note that deleting tensors using del operator works but you won't see a decrease in the GPU memory. Why? Because the memory is freed but not returned to the device. It is an optimization technique and from the user's perspective, the memory has been "freed". That is, the memory is available for making new tensors now.

Hence, deleting tensors is not recommended to free up GPU memory.