Instruction: "Can't use matmul on the given tensors" error when converting pytorch to onnx JS

Input:
I made a simple pytorch MLP(GAN generator) and converted it to onnx using the tutorial (https://www.youtube.com/watch?v=Vs730jsRgO8), my code is a bit different but I cant catch the error.

class Generator(nn.Module):
def __init__(self, g_input_dim, g_output_dim):
    super(Generator, self).__init__()
    # g_input = 100
    self.net = nn.Sequential(
      nn.Linear(g_input_dim, 256),
      nn.LeakyReLU(.2),
      nn.Linear(256, 512),
      nn.LeakyReLU(.2),
      nn.Linear(512, 1024),
      nn.LeakyReLU(.2),
      nn.Linear(1024, 784),
      nn.Tanh()
    )

# forward method
def forward(self, x):
    return self.net(x)

After training I export the model to onnx.

torch.save(G.state_dict(), "pytorch_model.pth")
import torch.onnx

model = Generator(z_dim,mnist_dim)
state_dict = torch.load("pytorch_model.pth")

model.load_state_dict(state_dict)
model.eval()

dummy_input = torch.zeros(100)

torch.onnx.export(model, dummy_input, "onnx_model.onnx", verbose=True)

Which gives the following onnx graph, which seems accurate.

graph(%input.1 : Float(100),
      %net.0.bias : Float(256),
      %net.2.bias : Float(512),
      %net.4.bias : Float(1024),
      %net.6.bias : Float(784),
      %25 : Float(100, 256),
      %26 : Float(256, 512),
      %27 : Float(512, 1024),
      %28 : Float(1024, 784)):
  %10 : Float(256) = onnx::MatMul(%input.1, %25) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1612:0
  %11 : Float(256) = onnx::Add(%10, %net.0.bias)
  %12 : Float(256) = onnx::LeakyRelu[alpha=0.20000000000000001](%11) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1239:0
  %14 : Float(512) = onnx::MatMul(%12, %26) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1612:0
  %15 : Float(512) = onnx::Add(%14, %net.2.bias)
  %16 : Float(512) = onnx::LeakyRelu[alpha=0.20000000000000001](%15) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1239:0
  %18 : Float(1024) = onnx::MatMul(%16, %27) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1612:0
  %19 : Float(1024) = onnx::Add(%18, %net.4.bias)
  %20 : Float(1024) = onnx::LeakyRelu[alpha=0.20000000000000001](%19) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1239:0
  %22 : Float(784) = onnx::MatMul(%20, %28) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1612:0
  %23 : Float(784) = onnx::Add(%22, %net.6.bias)
  %24 : Float(784)

Then I imported the code into javascript.

<html>
  <body>
    <script src="./onnx.min.js"></script>
    <script>
      async function test() {
        const sess = new onnx.InferenceSession()
        await sess.loadModel('./onnx_model.onnx')
        const input = new onnx.Tensor(new Float32Array(100), 'float32', [100])
        const outputMap = await sess.run([input])
        const outputTensor = outputMap.values().next().value
        console.log(`Output tensor: ${outputTensor.data}`)
      }
      test()
    </script>
  </body>
</html>

I know the input dimension is correct but onnx gives me the following error.

onnx.min.js:8 Uncaught (in promise) Error: Can't use matmul on the given tensors
    at e.createProgramInfo (onnx.min.js:8)
    at t.run (onnx.min.js:8)
    at e.run (onnx.min.js:8)
    at t.<anonymous> (onnx.min.js:14)
    at onnx.min.js:14
    at Object.next (onnx.min.js:14)
    at onnx.min.js:14
    at new Promise (<anonymous>)
    at r (onnx.min.js:14)
    at onnx.min.js:14

I also know that matmul is a supported operator with onnx, but I can't figure out how or if my input tensor is correct.

Output:
I think the matmul operator expects the input to be 2 dimensional. It seems to work when I add a batch size dimension to the input (a batch size of 1):

Before: dummy_input = torch.zeros(100)

After: dummy_input = torch.zeros(1, 100)

Before: const input = new onnx.Tensor(new Float32Array(100), 'float32', [100])

After: const input = new onnx.Tensor(new Float32Array(100), 'float32', [1, 100])


Instruction: Change input shape dimensions for ResNet model (pytorch)

Input:
I want to feed my 3,320,320 pictures in an existing ResNet model. The model actually expects input of size 3,32,32. As I am afraid of loosing information I don't simply want to resize my pictures.
What is the best way to preprocess my images, so that they are able to run on the ResNet34?
Should I add additional layers in the forward method of ResNet? If yes, what would be a suitable combination in my case?

import torch
import torch.nn as nn
import torch.nn.functional as F
from pytorch_fitmodule import FitModule
from torch.autograd import Variable
import numpy as np

def conv3x3(in_planes, out_planes, stride=1):
    return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False)

class BasicBlock(FitModule):
    expansion = 1

    def __init__(self, in_planes, planes, stride=1):
        super(BasicBlock, self).__init__()
        self.conv1 = conv3x3(in_planes, planes, stride)
        self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = conv3x3(planes, planes)
        self.bn2 = nn.BatchNorm2d(planes)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != self.expansion * planes:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_planes, self.expansion * planes,
                          kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(self.expansion * planes)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)
        out = F.relu(out)
        return out

class ResNet(FitModule):
    def __init__(self, block, num_blocks, num_classes=10):
        super(ResNet, self).__init__()
        self.in_planes = 64

        self.conv1 = conv3x3(3, 64)
        self.bn1 = nn.BatchNorm2d(64)
        self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
        self.linear = nn.Linear(512 * block.expansion, num_classes)

    def _make_layer(self, block, planes, num_blocks, stride):
        strides = [stride] + [1] * (num_blocks - 1)
        layers = []
        for stride in strides:
            layers.append(block(self.in_planes, planes, stride))
            self.in_planes = planes * block.expansion
        return nn.Sequential(*layers)

    def forward(self, x):    # add additional layers here?
        x = x.float()
        out = F.relu(self.bn1(self.conv1(x).float()).float())
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = F.avg_pool2d(out, 4)
        out = out.view(out.size(0), -1)
        out = self.linear(out)
        return out

def ResNet34():
    return ResNet(BasicBlock, [3, 4, 6, 3])

Thanks plenty!

Regards,
Fabian

Output:
If you change your avg_pool operation to 'AdaptiveAvgPool2d' your model will work for any image size.

However with your current setup, your 320x320 images would be 40x40 going into the pooling stage, which is a large feature map to pool over. Consider adding more conv layers.


Instruction: How to mask a 3D tensor with 2D mask and keep the dimensions of original vector?

Input:
Suppose, I have a 3D tensor A

A = torch.arange(24).view(4, 3, 2)
print(A)

and require masking it using 2D tensor

mask = torch.zeros((4, 3), dtype=torch.int64)  # or dtype=torch.ByteTensor
mask[0, 0] = 1
mask[1, 1] = 1
mask[3, 0] = 1
print('Mask: ', mask)

Using masked_select functionality from PyTorch leads to the following error.

torch.masked_select(X, (mask == 1))

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-72-fd6809d2c4cc> in <module>
     12
     13 # Select based on new mask
---> 14 Y = torch.masked_select(X, (mask == 1))
     15 #Y = X * mask_
     16 print(Y)

RuntimeError: The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 2

How to mask a 3D tensor with a 2D mask and keep the dimensions of the original vector? Any hints will be appreciated.

Output:
Essentially, we need to match the dimension of the tensor mask with the tensor being masked.

There are two ways to do it.

Approach 1: Does not preserve original tensor dimensions.

X = torch.arange(24).view(4, 3, 2)
print(X)

mask = torch.zeros((4, 3), dtype=torch.int64)  # or dtype=torch.ByteTensor
mask[0, 0] = 1
mask[1, 1] = 1
mask[3, 0] = 1
print('Mask: ', mask)

# Add a dimension to the mask tensor and expand it to the size of original tensor
mask_ = mask.unsqueeze(-1).expand(X.size())
print(mask_)

# Select based on the new expanded mask
Y = torch.masked_select(X, (mask_ == 1)) # does not preserve the dims
print(Y)

The output for approach 1:

tensor([ 0,  1,  8,  9, 18, 19])

Approach 2: Preserves the original tensor dimensions (by padding).

X = torch.arange(24).view(4, 3, 2)
print(X)

mask = torch.zeros((4, 3), dtype=torch.int64)  # or dtype=torch.ByteTensor
mask[0, 0] = 1
mask[1, 1] = 1
mask[3, 0] = 1
print('Mask: ', mask)

# Add a dimension to the mask tensor and expand it to the size of original tensor
mask_ = mask.unsqueeze(-1).expand(X.size())
print(mask_)

# Select based on the new expanded mask
Y = X * mask_
print(Y)

The output for approach 2:

tensor([[[ 0,  1],
         [ 2,  3],
         [ 4,  5]],

        [[ 6,  7],
         [ 8,  9],
         [10, 11]],

        [[12, 13],
         [14, 15],
         [16, 17]],

        [[18, 19],
         [20, 21],
         [22, 23]]])
Mask:  tensor([[1, 0, 0],
        [0, 1, 0],
        [0, 0, 0],
        [1, 0, 0]])
tensor([[[1, 1],
         [0, 0],
         [0, 0]],

        [[0, 0],
         [1, 1],
         [0, 0]],

        [[0, 0],
         [0, 0],
         [0, 0]],

        [[1, 1],
         [0, 0],
         [0, 0]]])
tensor([[[ 0,  1],
         [ 0,  0],
         [ 0,  0]],

        [[ 0,  0],
         [ 8,  9],
         [ 0,  0]],

        [[ 0,  0],
         [ 0,  0],
         [ 0,  0]],

        [[18, 19],
         [ 0,  0],
         [ 0,  0]]]


Instruction: concatenating two tensors in pytorch(with a twist)

Input:
I have a word embedding (of type tensor)of size torch.Size([8, 768]) stored in the variable embeddings which looks something like this :-

 tensor([[-0.0687, -0.1327,  0.0112,  ...,  0.0715, -0.0297, -0.0477],
        [ 0.0115, -0.0029,  0.0323,  ...,  0.0277, -0.0297, -0.0599],
        [ 0.0760,  0.0788,  0.1640,  ...,  0.0574, -0.0805,  0.0066],
        ...,
        [-0.0110, -0.1773,  0.1143,  ...,  0.1397,  0.3021,  0.1670],
        [-0.1379, -0.0294, -0.0026,  ..., -0.0966, -0.0726,  0.1160],
        [ 0.0466, -0.0113,  0.0283,  ..., -0.0735,  0.0496,  0.0963]],
       grad_fn=<IndexBackward>)

Now, I wish to take mean of some embeddings and place the mean back in the tensor.
For example,(I'll explain with the help of list and not tensor)

a = [1,2,3,4,5]
output = [1.5, 3, 4, 5]

So, here I have taken mean of 1 and 2 and then placed in the list output by shifting the elements to the  left in the list. I want to do the same thing for tensors as well.

I have the index stored in variable i from where I need to take average and j variable is being used for the stopping index. Now, let's look into the code :-

if i != len(embeddings):
  sum = 0
  count = 0
  #Calculating sum
  for x in range(i-1, j):
    sum += text_index[x]
    count += 1

  avg = sum/count

  #Inserting the average in place of the other embeddings
  embeddings = embeddings[:i-1] + [avg] + embeddings[j:]
else :
  pass

Now, I am getting an error at this line embeddings = embeddings[:i-1] + [avg] + embeddings[j:]
The error is :-

TypeError: unsupported operand type(s) for +: 'Tensor' and 'list'

Now, I understand that the above code would have worked well if embeddings was a list but it is a tensor. How do I do it?

NOTE :

*1. *embeddings.shape : torch.Size([8, 768])
 2.  avg is of type float**

Output:
To concatenate multiple tensors you can use torch.cat, where the list of tensors are concatenate across the specified dimensions. That requires that all tensors have the same number of dimensions and all dimensions except the one that they are concatenated on, need to have the same size.

Your embeddings has size [8, 768], therefore the left part and the right part will have size [num_left, 768] and [num_right, 768] respectively. And avg has size [768] (it's a tensor, not a single float) since you averaged multiple embeddings into one. In order to concatenate them with the two other parts, it needs to have size [1, 768], so that it can be concatenated on the first dimension to create a tensor of size [num_left + 1 + num_right, 768]. The singular first dimension can be added with torch.unsqueeze.

embeddings = torch.cat([embeddings[:i-1], avg.unsqueeze(0), embeddings[j:]], dim=0)

The for loop can also be replaced by slicing the tensor and taking the mean with torch.mean.

# keepdim=True keeps the dimension that the average is taken on
# So the output has size [1, 768] instead of [768]
avg = torch.mean(embeddings[i-1:j], dim=0, keepdim=True)

embeddings = torch.cat([embeddings[:i-1], avg, embeddings[j:]], dim=0)


Instruction: Using LSTM stateful for passing context b/w batches; may be some error in context passing, not getting good results?

Input:
I have checked the data before giving it to the network. The data is correct.
Using LSTM and passing the context b/w batches. per_class_accuracy is changing, but the loss is not going down. Been stuck for long, not sure if there is an error in the Code?
I have multi-class classification problem based upon an imbalanced dataset
Dataset_type: CSV
Dataset_size: 20000
Based upon CSV data of sensors
X = 0.6986111111111111,0,0,1,0,1,0,0,0,1,0,0,0,0,1,0,0,0,1,1,0,0,0
Y = leaveHouse
Per class accuracy:
{'leaveHouse': 0.34932855, 'getDressed': 1.0, 'idle': 0.8074534, 'prepareBreakfast': 0.8, 'goToBed': 0.35583413, 'getDrink': 0.0, 'takeShower': 1.0, 'useToilet': 0.0, 'eatBreakfast': 0.8857143}
Training:
# Using loss weights, the inverse of class frequency

criterion = nn.CrossEntropyLoss(weight = class_weights)

 hn, cn = model.init_hidden(batch_size)
            for i, (input, label) in enumerate(trainLoader):
                hn.detach_()
                cn.detach_()
                input = input.view(-1, seq_dim, input_dim)

                if torch.cuda.is_available():
                    input = input.float().cuda()
                    label = label.cuda()
                else:
                    input = input.float()
                    label = label

                # Forward pass to get output/logits
                output, (hn, cn) = model((input, (hn, cn)))

                # Calculate Loss: softmax --> cross entropy loss
                loss = criterion(output, label)#weig pram
                running_loss += loss
                loss.backward()  # Backward pass
                optimizer.step()  # Now we can do an optimizer step
                optimizer.zero_grad()  # Reset gradients tensors

Network

class LSTMModel(nn.Module):
    def init_hidden(self, batch_size):
        self.batch_size = batch_size
        if torch.cuda.is_available():
            hn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim).cuda()
            # Initialize cell state
            cn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim).cuda()
        else:
            hn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim)
            # Initialize cell state
            cn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim)
        return hn, cn

    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, seq_dim):
        super(LSTMModel, self).__init__()
        # Hidden dimensions
        self.hidden_dim = hidden_dim

        # Number of hidden layers
        self.layer_dim = layer_dim

        self.input_dim = input_dim
        # Building your LSTM
        # batch_first=True causes input/output tensors to be of shape
        # (batch_dim, seq_dim, feature_dim)
        self.lstm = nn.LSTM(self.input_dim, hidden_dim, layer_dim, batch_first=True)

        # Readout layer
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        self.seq_dim = seq_dim

    def forward(self, inputs):
        # Initialize hidden state with zeros
        input, (hn, cn) = inputs
        input = input.view(-1, self.seq_dim, self.input_dim)

        # time steps
        out, (hn, cn) = self.lstm(input, (hn, cn))

        # Index hidden state of last time step
        out = self.fc(out[:, -1, :])
        out = self.softmax(out)
        return out, (hn,cn)

Output:
One problem you might have is CrossEntropyLoss combines a log softmax operation with negative log likelihood loss, but you're applying a softmax in your model. You should pass the raw logits out of the final layer to CrossEntropyLoss.

Also I an't say without seeing the models forward pass, but it looks like you're applying the softmax on dimension 1 to a tensor that (I'm inferring) has shape batch_size, sequence_length, output_dim, when you should be applying it along the output dim.


Instruction: Does the reduction of the dimensions over multiple layers allow more details to be stored within the final representation?

Input:
From : https://debuggercafe.com/implementing-deep-autoencoder-in-pytorch/
the following autoencoder is defined

class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()

        # encoder
        self.enc1 = nn.Linear(in_features=784, out_features=256)
        self.enc2 = nn.Linear(in_features=256, out_features=128)
        self.enc3 = nn.Linear(in_features=128, out_features=64)
        self.enc4 = nn.Linear(in_features=64, out_features=32)
        self.enc5 = nn.Linear(in_features=32, out_features=16)

        # decoder
        self.dec1 = nn.Linear(in_features=16, out_features=32)
        self.dec2 = nn.Linear(in_features=32, out_features=64)
        self.dec3 = nn.Linear(in_features=64, out_features=128)
        self.dec4 = nn.Linear(in_features=128, out_features=256)
        self.dec5 = nn.Linear(in_features=256, out_features=784)

    def forward(self, x):
        x = F.relu(self.enc1(x))
        x = F.relu(self.enc2(x))
        x = F.relu(self.enc3(x))
        x = F.relu(self.enc4(x))
        x = F.relu(self.enc5(x))

        x = F.relu(self.dec1(x))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        x = F.relu(self.dec5(x))
        return x

net = Autoencoder()

From the Autoencoder class, we can see that 784 features are passed through a series of transformations and are converted to 16 features.

The transformations (in_features to out_features) for each layer are:

784 to 256
256 to 128
128 to 64
64 to 32
32 to 16

Why do we perform this sequence of operations? For example, why don't we perform the following sequence of operations instead?

784 to 256
256 to 128

Or maybe

784 to 512
512 to 256
256 to 128

Or maybe just encode in two layers:

784 to 16

Does the reduction of the dimensions over multiple layers (instead of a single layer) allow more details to be stored within the final representation? For example, if we used only the transformation $784 \rightarrow 16$, may this cause some detail not to be encoded? If so, why is this the case?

Output:
Well yes. Reduction of dimensions over multiple layers allow the model to learn more information than it would otherwise. You could do an experiment with it too; change the autoencoder to be much shallower and see what representation it learns!

In any case, think of it in simpler terms: Let's say you have a regression system. As opposed to a simple linear transformation like y = a0 + a1.x, a regression that uses higher degree polynomials to approximate the task has much more approximation power, for a lack of better terms. Which is why they tend to overfit easily; they're approximating the training data too perfectly and thus have trouble generalizing later. However, as a rule of thumb, an overfitting model may be considered as a more preferable scenario than an underfitting one: because you can regularize the overfitting model to generalize better, but the underfitting model may simply not be capable of approximating your task.

Now let's think of a deep neural network. At it's core, a neural network is also a highly complicated function that helps you approximate a task. If you have a linear layer that maps 784 input features to 16 output features, it's basically doing a simple transformation: ReLU(X.W+B). But if you're passing through multiple layers, your transformation becomes something like ReLU((ReLU((ReLU(X.W1+B1)).W2+B2)).W3+B3) and so on. It's a much more complex function with higher approximation power, which means it's liable to better approximate your task, and learn a better representation. This concept holds true for Deep Learning in general, and even truer for convolutional layers in particular. You'll notice most deep learning model architectures are basically layers or blocks repeatedly stacked one after another.


Instruction: PyTorch tensor slice and memory usage

Input:
import torch
T = torch.FloatTensor(range(0,10 ** 6)) # 1M

#case 1:
torch.save(T, 'junk.pt')
# results in a 4 MB file size

#case 2:
torch.save(T[-20:], 'junk2.pt')
# results in a 4 MB file size

#case 3:
torch.save(torch.FloatTensor(T[-20:]), 'junk3.pt')
# results in a 4 MB file size

#case 4:
torch.save(torch.FloatTensor(T[-20:].tolist()), 'junk4.pt')
# results in a 405 Bytes file size

My questions are:

In case 3 the resulting file size seems surprising as we are creating a new tensor. Why is this new tensor not just the slice?

Is case 4, the optimal method for saving just part (slice) of a tensor?

More generally, if I want to 'trim' a very large 1-dimensional tensor by removing the first half of its values in order to save memory, do I have to proceed as in case 4, or is there a more direct and less computationally costly way that does not involve creating a python list.

Output:
  (i) In case 3 the resulting file size seems surprising as we are creating a new tensor. Why is this new tensor not just the slice?

Slicing creates a view of the tensor, which shares the underlying data but contains information about the memory offsets used for the visible data. This avoids having to copy the data frequently, which makes a lot of operations much more efficient. See PyTorch - Tensor Views for a list of affected operations.

You are dealing with one of the few cases, where the underlying data matters. To save the tensor, it needs to save the underlying data, otherwise the offsets would no longer be valid.

torch.FloatTensor does not create a copy of the tensor, if it's not necessary. You can verify that their underlying data is still the same (they have the exact same memory location):

torch.FloatTensor(T[-20:]).storage().data_ptr() == T.storage().data_ptr()
# => True

  (ii) Is case 4, the optimal method for saving just part (slice) of a tensor?

  (iii) More generally, if I want to 'trim' a very large 1-dimensional tensor by removing the first half of its values in order to save memory, do I have to proceed as in case 4, or is there a more direct and less computationally costly way that does not involve creating a python list.

You will most likely not get around copying the data of the slice, but at least you can avoid creating a Python list from it and creating a new tensor from the list, by using torch.Tensor.clone instead:

torch.save(T[-20:].clone(), 'junk5.pt')


Instruction: Passing embedded sequence to LSTM and getting TypeError: 'int' object is not subscriptable

Input:
I have some really basic pytorch code here where I'm trying to test running an input tensor through what will eventually become my forward function.

Goal: Treat the a sentence as a single input sequence after embedding each word number.

Embed a tensor
Convert that embedding back to a float32 tensor
Reshape embedding to shape (batch_size, seq_len, input_size)
Pass through lstm.

I've converted back to a float32 tensor after embedding so idk why I'm getting this error.

hidden_size=10
embedding = nn.Embedding(VOC.n_words, hidden_size)
lstm = nn.LSTM(hidden_size, hidden_size, # Will output 2x hidden size
               num_layers=1, dropout=0.5,
               bidirectional=True, batch_first=True)

print("Input tensor",idx_sentence)
# Forward test
embedded = embedding(idx_sentence.long())
embedded = torch.tensor(embedded, dtype=torch.float32)
print(f"embedding: {embedded.size()}")

# reshape to (batch_size, seq_len, input_size)
sequence = embedded.view(1,-1,hidden_size)
print(f"sequence shape: {sequence.size()}")

output, hidden = lstm(sequence, hidden_size)
print(f"output shape: {output.size()}")

Input tensor tensor([ 3., 20., 21., 90.,  9.])
embedding: torch.Size([5, 10])
sequence shape: torch.Size([1, 5, 10])
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:10: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  # Remove the CWD from sys.path while we load stuff.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-116-ab3d6ed0e51c> in <module>()
     16
     17 # Input have shape (seq_len, batch, input_size)
---> 18 output, hidden = lstm(sequence, hidden_size)
     19 print(f"output shape: {output.size()}")

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    520         expected_hidden_size = self.get_expected_hidden_size(input, batch_sizes)
    521
--> 522         self.check_hidden_size(hidden[0], expected_hidden_size,
    523                                'Expected hidden[0] size {}, got {}')
    524         self.check_hidden_size(hidden[1], expected_hidden_size,

TypeError: 'int' object is not subscriptable

Output:
The LSTM accepts two inputs, as described in nn.LSTM - Inputs:

input: the input sequences
(h_0, c_0): a tuple with the initial hidden state h_0 and the initial cell state c_0.

But you are passing hidden_size as the second argument, which is an int and not a tuple. When the tuple is unpacked it fails, since hidden_size[0] does not work, as integers cannot be indexed.

The second argument is optional and if you don't provide it, the hidden and cell states will default to zero. That's usually what you want, therefore you can leave it off:

output, hidden = lstm(sequence)


Instruction: Generating pytorch's theta from affine transform matrix

Input:
I'm trying to perform a rigid + scale transformation on a 3D volume with pytorch, but I can't seem to understand how the theta required for torch.nn.functional.affine_grid works.

I have a transformation matrix of size (1,4,4) generated by multiplying the matrices Translation * Scale * Rotation. If I use this matrix in, for example, scipy.ndimage.affine_transform, it works with no issues. However, the same matrix (cropped to size (1,3,4)) fails completely with torch.nn.functional.affine_grid.

I have managed to understand how the translation works (range -1 to 1) and I have confirmed that the Translation matrix works by simply normalizing the values to that range. As for the other two, I am lost.

I tried using a basic Scaling matrix alone (below) as a most basic comparison but the results in pytorch are different than that of scipy

Scaling =
[[0.75, 0, 0, 0],
[[0, 0.75, 0, 0],
[[0, 0, 0.75, 0],
[[0, 0, 0, 1]]

How can I convert the (1,4,4) affine matrix to work the same with torch.nn.functional.affine_grid? Alternatively, is there a way to generate the correct matrix based on the transformation parameters (shift, euler angles, scaling)?

Output:
To anyone that comes across a similar issue in the future, the problem with scipy vs pytorch affine transforms is that scipy applies the transforms around (0, 0, 0) while pytorch applies it around the middle of the image/volume.

For example, let's take the parameters:

euler_angles = [ea0, ea1, ea2]
translation = [tr0, tr1, tr2]
scale = [sc0, sc1, sc2]

and create the following transformation matrices:

# Rotation matrix
R_x(ea0, ea1, ea2) = np.array([[1, 0, 0, 0],
                              [0, math.cos(ea0), -math.sin(ea0), 0],
                              [0, math.sin(ea0), math.cos(ea0), 0],
                              [0, 0, 0, 1]])

R_y(ea0, ea1, ea2) = np.array([[math.cos(ea1), 0, math.sin(ea1), 0],
                               [0, 1, 0, 0],
                               [-math.sin(ea1), 0, math.cos(ea1)], 0],
                               [0, 0, 0, 1]])

R_z(ea0, ea1, ea2) = np.array([[math.cos(ea2), -math.sin(ea2), 0, 0],
                               [math.sin(ea2), math.cos(ea2), 0, 0],
                               [0, 0, 1, 0],
                               [0, 0, 0, 1]])

R = R_x.dot(R_y).dot(R_z)

# Translation matrix
T(tr0, tr1, tr2) = np.array([[1, 0, 0, -tr0],
                             [0, 1, 0, -tr1],
                             [0, 0, 1, -tr2],
                             [0, 0, 0, 1]])
# Scaling matrix
S(sc0, sc1, sc2) = np.array([[1/sc0, 0, 0, 0],
                             [0, 1/sc1, 0, 0],
                             [0, 0, 1/sc2, 0],
                             [0, 0, 0, 1]])

If you have a volume of size (100, 100, 100), the scipy transform around the centre of the volume requires moving the centre of the volume to (0, 0, 0) first, and then moving it back to (50, 50, 50) after S, T, and R have been applied. Defining:

T_zero = np.array([[1, 0, 0, 50],
                   [0, 1, 0, 50],
                   [0, 0, 1, 50],
                   [0, 0, 0, 1]])

T_centre = np.array([[1, 0, 0, -50],
                     [0, 1, 0, -50],
                     [0, 0, 1, -50],
                     [0, 0, 0, 1]])

The scipy transform around the centre is then:

transform_scipy_centre = T_zero.dot(T).dot(S).dot(R).T_centre

In pytorch, there are some slight differences to the parameters.

The translation is defined between -1 and 1. Their order is also different. Using the same (100, 100, 100) volume as an example, the translation parameters in pytorch are given by:

# Note the order difference
translation_pytorch = =[tr0_p, tr1_p, tr2_p] = [tr0/50, tr2/50, tr1/50]
T_p = T(tr0_p, tr1_p, tr2_p)

The scale parameters are in a different order:
    scale_pytorch = [sc0_p, sc1_p, sc2_p] = [sc2, sc0, sc1]
    S_p = S(sc0_p, sc1_p, sc2_p)

The euler angles are the biggest difference. To get the equivalent transform, first the parameters are negative and in a different order:

# Note the order difference
euler_angles_pytorch = [ea0_p, ea1_p, ea2_p] = [-ea0, -ea2, -ea1]
R_x_p = R_x(ea0_p, ea1_p, ea2_p)
R_y_p = R_y(ea0_p, ea1_p, ea2_p)
R_z_p = R_z(ea0_p, ea1_p, ea2_p)

The order in which the rotation matrix is calculated is also different:
    # Note the order difference
    R_p = R_x_p.dot(R_z_p).dot(R_y_p)

With all these considerations, the scipy transform with:

transform_scipy_centre = T_zero.dot(T).dot(S).dot(R).T_centre

is equivalent to the pytorch transform with:

transform_pytorch = T_p.dot(S_p).dot(R_p)

I hope this helps!


Instruction: ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index) throws IndexError: Target 42 is out of bounds

Input:
After loading CIFAR 100, I try to train my neural network. But I don't know why I get the out of bounds error shown below

Optimizing the network with batch size 25
Epoch: 0  of  30 Average loss: -
/mnt_home/klee/LBSBGenGapSharpnessResearch/vgg.py:43: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  return F.log_softmax(x)
Traceback (most recent call last):
  File "plot_parametric_pytorch_cifar100.py", line 130, in <module>
    loss_fn = F.nll_loss(ops, tgts)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 42 is out of bounds.

Here is the script I'm running:

cudnn.benchmark = True
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')
X_train = np.transpose(X_train, axes=(0, 3, 1, 2))
X_test = X_test.astype('float32')
X_test = np.transpose(X_test, axes=(0, 3, 1, 2))
X_train /= 255
X_test /= 255
device = torch.device('cuda:0')

# This is where you can load any model of your choice.
# I stole PyTorch Vision's VGG network and modified it to work on CIFAR-10.
# You can take this line out and add any other network and the code
# should run just fine.
model = vgg.vgg11_bn()
#model.to(device)

# Forward pass
opfun = lambda X: model.forward(Variable(torch.from_numpy(X)))

# Forward pass through the network given the input
predsfun = lambda op: np.argmax(op.data.numpy(), 1)

# Do the forward pass, then compute the accuracy
accfun = lambda op, y: np.mean(np.equal(predsfun(op), y.squeeze()))*100

# Initial point
x0 = deepcopy(model.state_dict())

# Number of epochs to train for
# Choose a large value since LB training needs higher values
# Changed from 150 to 30
nb_epochs = 30
batch_range = [25, 40, 50, 64, 80, 128, 256, 512, 625, 1024, 1250, 1750, 2048, 2500, 3125, 4096, 4500, 5000]

# parametric plot (i.e., don't train the network)
hotstart = False

if not hotstart:
    for batch_size in batch_range:
        optimizer = torch.optim.Adam(model.parameters())
        model.load_state_dict(x0)
        #model.to(device)
        average_loss_over_epoch = '-'
        print('Optimizing the network with batch size %d' % batch_size)
        np.random.seed(1337) #So that both networks see same sequence of batches
        for e in range(nb_epochs):
            model.eval()
            print('Epoch:', e, ' of ', nb_epochs, 'Average loss:', average_loss_over_epoch)
            average_loss_over_epoch = 0
            # Checkpoint the model every epoch
            torch.save(model.state_dict(), "./models/30EpochCIFAR100ExperimentBatchSize" + str(batch_size) + ".pth")
            array = np.random.permutation(range(X_train.shape[0]))
            slices = X_train.shape[0] // batch_size
            beginning = 0
            end = 1

            # Training loop!
            for _ in range(slices):
                start_index = batch_size * beginning
                end_index = batch_size * end
                smpl = array[start_index:end_index]
                model.train()
                optimizer.zero_grad()
                ops = opfun(X_train[smpl])
                tgts = Variable(torch.from_numpy(y_train[smpl]).long().squeeze())
                loss_fn = F.nll_loss(ops, tgts) <--- errorring linne
                average_loss_over_epoch += loss_fn.data.numpy() / (X_train.shape[0] // batch_size)
                loss_fn.backward()
                optimizer.step()
                beginning += 1
                end += 1

and here's the VGG model:

__all__ = [
    'VGG', 'vgg11', 'vgg11_bn', 'vgg13', 'vgg13_bn', 'vgg16', 'vgg16_bn',
    'vgg19_bn', 'vgg19',
]

model_urls = {
    'vgg11': 'https://s3.amazonaws.com/pytorch/models/vgg11-fb7e83b2.pth',
    'vgg13': 'https://s3.amazonaws.com/pytorch/models/vgg13-58758d87.pth',
    'vgg16': 'https://s3.amazonaws.com/pytorch/models/vgg16-82412952.pth',
    'vgg19': 'https://s3.amazonaws.com/pytorch/models/vgg19-341d7465.pth',
}

class VGG(nn.Module):
    def __init__(self, features):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Linear(4096, 10),
        )
        self._initialize_weights()

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return F.log_softmax(x)

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                if m.bias is not None:
                    m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()
            elif isinstance(m, nn.Linear):
                n = m.weight.size(1)
                m.weight.data.normal_(0, 0.01)
                m.bias.data.zero_()

def make_layers(cfg, batch_norm=False):
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = v
    return nn.Sequential(*layers)

cfg = {
    'A': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'B': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'E': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

def vgg11(pretrained=False, **kwargs):
    """VGG 11-layer model (configuration "A")

    Args:
        pretrained (bool): If True, returns a model pre-trained on ImageNet
    """
    model = VGG(make_layers(cfg['A']), **kwargs)
    if pretrained:
        model.load_state_dict(model_zoo.load_url(model_urls['vgg11']))
    return model

def vgg11_bn(**kwargs):
    """VGG 11-layer model (configuration "A") with batch normalization"""
    return VGG(make_layers(cfg['A'], batch_norm=True), **kwargs)

I'm not sure how to fix the index error. I think it has to do with the number of classes, but I'm  not sure where to fix that (in the above code): I read https://discuss.pytorch.org/t/indexerror-target-2-is-out-of-bounds/69614 but not sure where to go from here.

Output:
You are using CIFAR-100, which has 100 classes (hence the name). But your model only predicts 10 classes. Naturally, any class above 10 will result in an error.

The output of the last linear linear in the model's classifier needs to be changed to 100:

self.classifier = nn.Sequential(
    nn.Dropout(),
    nn.Linear(512, 4096),
    nn.ReLU(True),
    nn.Dropout(),
    nn.Linear(4096, 4096),
    nn.ReLU(True),
    nn.Linear(4096, 100),
)


Instruction: huggingface bert showing poor accuracy / f1 score [pytorch]

Input:
I am trying BertForSequenceClassification for a simple article classification task.

No matter how I train it (freeze all layers but the classification layer, all layers trainable, last k layers trainable), I always get an almost randomized accuracy score. My model doesn't go above 24-26% training accuracy (I only have 5 classes in my dataset).

I'm not sure what did I do wrong while designing/training the model. I tried the model with multiple datasets, every time it gives the same random baseline accuracy.

Dataset I used: BBC Articles (5 classes)

https://github.com/zabir-nabil/pytorch-nlp/tree/master/bbc

  Consists of 2225 documents from the BBC news website corresponding to
  stories in five topical areas from 2004-2005. Natural Classes: 5
  (business, entertainment, politics, sport, tech)

I added the model part and the training part which are the most important portion (to avoid any irrelevant details). I added the full source-code + data too if that's useful for reproducibility.

My guess is there is something wrong with the I way I designed the network or the way I'm passing the attention_masks/ labels to the model. Also, the token length 512 should not be a problem as most of the texts has length < 512 (the mean length is < 300).

Model code:

import torch
from torch import nn

class BertClassifier(nn.Module):
    def __init__(self):
        super(BertClassifier, self).__init__()
        self.bert = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 5)
        # as we have 5 classes

        # we want our output as probability so, in the evaluation mode, we'll pass the logits to a softmax layer
        self.softmax = torch.nn.Softmax(dim = 1) # last dimension
    def forward(self, x, attn_mask = None, labels = None):

        if self.training == True:
            # print(x.shape)
            loss = self.bert(x, attention_mask = attn_mask, labels = labels)
            # print(x[0].shape)

            return loss

        if self.training == False: # in evaluation mode
            x = self.bert(x)
            x = self.softmax(x[0])

            return x
    def freeze_layers(self, last_trainable = 1):
        # we freeze all the layers except the last classification layer + few transformer blocks
        for layer in list(self.bert.parameters())[:-last_trainable]:
            layer.requires_grad = False

# create our model

bertclassifier = BertClassifier()

Training code:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # cuda for gpu acceleration

# optimizer

optimizer = torch.optim.Adam(bertclassifier.parameters(), lr=0.001)

epochs = 15

bertclassifier.to(device) # taking the model to GPU if possible

# metrics

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

train_losses = []

train_metrics = {'acc': [], 'f1': []}
test_metrics = {'acc': [], 'f1': []}

# progress bar

from tqdm import tqdm_notebook

for e in tqdm_notebook(range(epochs)):
    train_loss = 0.0
    train_acc = 0.0
    train_f1 = 0.0
    batch_cnt = 0

    bertclassifier.train()

    print(f'epoch: {e+1}')

    for i_batch, (X, X_mask, y) in tqdm_notebook(enumerate(bbc_dataloader_train)):
        X = X.to(device)
        X_mask = X_mask.to(device)
        y = y.to(device)

        optimizer.zero_grad()

        loss, y_pred = bertclassifier(X, X_mask, y)

        train_loss += loss.item()
        loss.backward()
        optimizer.step()

        y_pred = torch.argmax(y_pred, dim = -1)

        # update metrics
        train_acc += accuracy_score(y.cpu().detach().numpy(), y_pred.cpu().detach().numpy())
        train_f1 += f1_score(y.cpu().detach().numpy(), y_pred.cpu().detach().numpy(), average = 'micro')
        batch_cnt += 1

    print(f'train loss: {train_loss/batch_cnt}')
    train_losses.append(train_loss/batch_cnt)
    train_metrics['acc'].append(train_acc/batch_cnt)
    train_metrics['f1'].append(train_f1/batch_cnt)

    test_loss = 0.0
    test_acc = 0.0
    test_f1 = 0.0
    batch_cnt = 0

    bertclassifier.eval()
    with torch.no_grad():
        for i_batch, (X, y) in enumerate(bbc_dataloader_test):
            X = X.to(device)
            y = y.to(device)

            y_pred = bertclassifier(X) # in eval model we get the softmax output so, don't need to index

            y_pred = torch.argmax(y_pred, dim = -1)

            # update metrics
            test_acc += accuracy_score(y.cpu().detach().numpy(), y_pred.cpu().detach().numpy())
            test_f1 += f1_score(y.cpu().detach().numpy(), y_pred.cpu().detach().numpy(), average = 'micro')
            batch_cnt += 1

    test_metrics['acc'].append(test_acc/batch_cnt)
    test_metrics['f1'].append(test_f1/batch_cnt)

Full source-code with the dataset is available here: https://github.com/zabir-nabil/pytorch-nlp/blob/master/bert-article-classification.ipynb

Update:

After observing the prediction, it seems model almost always predicts 0:

bertclassifier.eval()
with torch.no_grad():
    for i_batch, (X, y) in enumerate(bbc_dataloader_test):
        X = X.to(device)
        y = y.to(device)

        y_pred = bertclassifier(X) # in eval model we get the softmax output so, don't need to index

        y_pred = torch.argmax(y_pred, dim = -1)

        print(y)
        print(y_pred)
        print('--------------------')

tensor([4, 2, 2, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 3, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 0, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 4, 4, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 3, 2, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 3, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 1, 4, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 0, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 1, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 2, 4, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 1, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 1, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 0, 1, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 3, 1, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 2, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 1, 2, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 4, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 0, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 3, 2, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 1, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 2, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 3, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 4, 2, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 4, 4, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 1, 3, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 2, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 0, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 1, 4, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 4, 3, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 2, 1, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 4, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 1, 1, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 2, 4, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 2, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 1, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 2, 2, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 2, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 3, 2, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 0, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 1, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 4, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 3, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 2, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 2, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 2, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 2, 3, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 3, 0, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 0, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 2, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 4, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 0, 4, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 2, 0, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 1, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 1, 3, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 3, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 2, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 0, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 0, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 1, 1, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 1, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 4, 1, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 2, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 4, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 4, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 1, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 4, 3, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 3, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 0, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 1, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 2, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 4, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 2, 4, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
...
...

Actually, the model is always predicting the same output [0.2270, 0.1855, 0.2131, 0.1877, 0.1867] for any input, it's like it didn't learn anything at all.

It's weird because my dataset is not imbalanced.

Counter({'politics': 417,
         'business': 510,
         'entertainment': 386,
         'tech': 401,
         'sport': 511})

Output:
After some digging I found out, the main culprit was the learning rate, for fine-tuning bert 0.001 is extremely high. When I reduced my learning rate from 0.001 to 1e-5, both my training and test accuracy reached 95%.

  When BERT is fine-tuned, all layers are trained - this is quite different from fine-tuning in a lot of other ML models, but it matches what was described in the paper and works quite well (as long as you only fine-tune for a few epochs - it's very easy to overfit if you fine-tune the whole model for a long time on a small amount of data!)

src: https://github.com/huggingface/transformers/issues/587

  Best result is found when all the layers are trained with a really small learning rate.

src: https://github.com/uzaymacar/comparatively-finetuning-bert


Instruction: How to put image uploaded in tkinter into a function?

Input:
I am trying to create a Python tkinter application where the user can upload an image from file and the image is put through a image segmentation function which outputs an matplotlib plot.

I have the image segmentation function, it takes two parameters: neural network, image file pathway.

from torchvision import models
from PIL import Image
import matplotlib.pyplot as plt
import torch
import torchvision.transforms as T
import numpy as np

fcn = models.segmentation.fcn_resnet101(pretrained=True).eval()

# Define the helper function
def decode_segmap(image, nc=21):

  label_colors = np.array([(0, 0, 0),  # 0=background
               # 1=aeroplane, 2=bicycle, 3=bird, 4=boat, 5=bottle
               (128, 0, 0), (0, 128, 0), (128, 128, 0), (0, 0, 128), (128, 0, 128),
               # 6=bus, 7=car, 8=cat, 9=chair, 10=cow
               (0, 128, 128), (128, 128, 128), (64, 0, 0), (192, 0, 0), (64, 128, 0),
               # 11=dining table, 12=dog, 13=horse, 14=motorbike, 15=person
               (192, 128, 0), (64, 0, 128), (192, 0, 128), (64, 128, 128), (192, 128, 128),
               # 16=potted plant, 17=sheep, 18=sofa, 19=train, 20=tv/monitor
               (0, 64, 0), (128, 64, 0), (0, 192, 0), (128, 192, 0), (0, 64, 128)])

  r = np.zeros_like(image).astype(np.uint8)
  g = np.zeros_like(image).astype(np.uint8)
  b = np.zeros_like(image).astype(np.uint8)

  for l in range(0, nc):
    idx = image == l
    r[idx] = label_colors[l, 0]
    g[idx] = label_colors[l, 1]
    b[idx] = label_colors[l, 2]

  rgb = np.stack([r, g, b], axis=2)
  return rgb

def segment(net, path):
  img = Image.open(path)
  plt.imshow(img); plt.axis('off'); plt.show()
  # Comment the Resize and CenterCrop for better inference results
  trf = T.Compose([T.Resize(256),
                   T.CenterCrop(224),
                   T.ToTensor(),
                   T.Normalize(mean = [0.485, 0.456, 0.406],
                               std = [0.229, 0.224, 0.225])])
  inp = trf(img).unsqueeze(0)
  out = net(inp)['out']
  om = torch.argmax(out.squeeze(), dim=0).detach().cpu().numpy()
  rgb = decode_segmap(om)
  plt.imshow(rgb); plt.axis('off'); plt.show()

I have not made the Tkinter GUI as I am unsure how I can take the image uploaded from file and convert it into a file path (string) and put it through the function. There is only one neural network for now, being fcn.

Output:
Firstly import filedialog and PIL:
from tkinter import filedialog
from PIL import Image

Now use a variable path (or anything) to define the path that is returned when you choose inside of a GUI.
path = filedialog.askopenfilename(initialdir='/Downloads', title='Select Photo', filetypes=(('JPEG files', '*.jpg'), ('PNG files', '*.png')))

You can use any filetype you want by specifying it in the argument there with the title and '*.extension'. NOTE THAT IT IS CASE-SENSITIVE. Now your variable path has the path for the image and all you have to do is open it with PIL using
img = Image.open(path)

Just like you did in your code


Instruction: L-BFGS-B code, Scipy (sciopt.fmin_l_bfgs_b(func, init_guess, maxiter=10, bounds=list(bounds), disp=1, iprint=101))

Input:
I'm using the L-BFGS-B optimizer to find the minima of a function. This will help me calculate sharpness for the function. However, I'm not sure if this following message is considered a normal message (i.e. Is there something wrong with my program or is this message typical?) See below:

RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =     28149514     M =           10

At X0         0 variables are exactly at the bounds
^[[C
At iterate    0    f= -3.59325D+00    |proj g|=  2.10249D-03

At iterate    1    f= -2.47853D+01    |proj g|=  4.20499D-03

 Bad direction in the line search;
   refresh the lbfgs memory and restart the iteration.

At iterate    2    f= -2.53202D+01    |proj g|=  4.17686D-03

At iterate    3    f= -2.53202D+01    |proj g|=  4.17686D-03

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
*****      3     43 ******     0 *****   4.177D-03  -2.532D+01
  F =  -25.320247650146484

CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH

 Warning:  more than 10 function and gradient
   evaluations in the last line search.  Termination
   may possibly be caused by a bad search direction.

I got the following sharpness anyway which is relatively consistent with the paper I'm trying to reproduce: It's just that I'm a bit concerned with the above message.
tensor(473.0201)

Here is my code for computing sharpness:

def get_sharpness(data_loader, model, criterion, epsilon, manifolds=0):

  # extract current x0
  x0 = None
  for p in model.parameters():
    if x0 is None:
      x0 = p.data.view(-1)
    else:
      x0 = torch.cat((x0, p.data.view(-1)))
  x0 = x0.cpu().numpy()

  # get current f_x
  f_x0, _ = get_minus_cross_entropy(x0, data_loader, model, criterion)
  f_x0 = -f_x0
  logging.info('min loss f_x0 = {loss:.4f}'.format(loss=f_x0))

  # find the minimum
  if 0==manifolds:
    x_min = np.reshape(x0 - epsilon * (np.abs(x0) + 1), (x0.shape[0], 1))
    x_max = np.reshape(x0 + epsilon * (np.abs(x0) + 1), (x0.shape[0], 1))
    bounds = np.concatenate([x_min, x_max], 1)
    func = lambda x: get_minus_cross_entropy(x, data_loader, model, criterion, training=True)
    init_guess = x0
  else:
    warnings.warn("Small manifolds may not be able to explore the space.")
    assert(manifolds<=x0.shape[0])
    #transformer = rp.GaussianRandomProjection(n_components=manifolds)
    #transformer.fit(np.random.rand(manifolds, x0.shape[0]))
    #A_plus = transformer.components_
    #A = np.linalg.pinv(A_plus)
    A_plus = np.random.rand(manifolds, x0.shape[0])*2.-1.
    # normalize each column to unit length
    A_plus_norm = np.linalg.norm(A_plus, axis=1)
    A_plus = A_plus / np.reshape(A_plus_norm, (manifolds,1))
    A = np.linalg.pinv(A_plus)
    abs_bound = epsilon * (np.abs(np.dot(A_plus, x0))+1)
    abs_bound = np.reshape(abs_bound, (abs_bound.shape[0], 1))
    bounds = np.concatenate([-abs_bound, abs_bound], 1)
    def func(y):
      floss, fg = get_minus_cross_entropy(x0 + np.dot(A, y), data_loader, model, criterion, training=True)
      return floss, np.dot(np.transpose(A), fg)
    #func = lambda y: get_minus_cross_entropy(x0+np.dot(A, y), data_loader, model, criterion, training=True)
    init_guess = np.zeros(manifolds)

  #rand_selections = (np.random.rand(bounds.shape[0])+1e-6)*0.99
  #init_guess = np.multiply(1.-rand_selections, bounds[:,0])+np.multiply(rand_selections, bounds[:,1])

  minimum_x, f_x, d = sciopt.fmin_l_bfgs_b(func, init_guess, maxiter=10, bounds=list(bounds), disp=1, iprint=101)
    #factr=10.,
    #pgtol=1.e-12,

  f_x = -f_x
  logging.info('max loss f_x = {loss:.4f}'.format(loss=f_x))
  sharpness = (f_x - f_x0)/(1+f_x0)*100
  print(sharpness)

  # recover the model
  x0 = torch.from_numpy(x0).float()
  x0 = x0.cuda()
  x_start = 0
  for p in model.parameters():
      psize = p.data.size()
      peltnum = 1
      for s in psize:
          peltnum *= s
      x_part = x0[x_start:x_start + peltnum]
      p.data = x_part.view(psize)
      x_start += peltnum

  return sharpness

Which was taken from this repository:
https://github.com/wenwei202/smoothout/blob/master/measure_sharpness.py

I'm concerned about exact accuracy.

Output:
First, l-bfgs-b will only give a global minimum for a convex function.
the message
CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH
is the normal convergence message.
The warning you are getting says that there are a lot of function/gradient evaluations in the line search - this can often happen when you use l-bfgs-b on non convex functions. So if the thing you're minimizing is non convex (and it seems like it might be just by glancing at the code), I would say this is normal.


Instruction: How can I keep a PyTorch submodule in eval mode?

Input:
I have a pretrained model that I'm using in conjunction with a model being trained. I want the pretrained model to always be in eval mode, but the other model will be moving back and forth between eval and train mode. I'd still like the pretrained model to be a submodule of the other one, though (e.g. so that all parameters stay on the same device). Is there a way to do this? Here's a minimal example:

from torch import nn

class FixedModule(nn.Module):
    pass

class TrainableModule(nn.Module):
    def __init__(self, fixed_module):
        super().__init__()
        self.fixed_module = fixed_module

fixed = FixedModule().eval()
assert not fixed.training

trainable = TrainableModule(fixed)
assert trainable.training and not trainable.fixed_module.training

trainable.train()
assert trainable.fixed_module.training  # I'd like this to give an error

I know I can work around this by, e.g., always doing

trainable.train()
trainable.fixed_module.eval()

but that's error-prone and doesn't work well with existing code.

Output:
You can override train in FixedModule to prevent it from changing modes. Note that eval just calls train(False), so you don't need to override that too. But calling FixedModule.eval won't do anything now, so you have to set training = False in init.

from torch import nn

class FixedModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.training = False

        # add any other nn.Module attributes here before calling self.children

        # you could override `train` in each child too if you really wanted,
        # but that seems like overkill unless there are external references
        # to any submodules of FixedModule
        for module in self.children():
            module.eval()

    def train(self, mode):
        return self

class TrainableModule(nn.Module):
    def __init__(self, fixed_module):
        super().__init__()
        self.fixed_module = fixed_module

fixed = FixedModule().eval()
assert not fixed.training

trainable = TrainableModule(fixed)
assert trainable.training and not trainable.fixed_module.training

trainable.train()
assert not trainable.fixed_module.training # passes


Instruction: Is maxpooling on odd number possible?

Input:
I am going through the Udacity DeepLearning Nanodegree and working on the autoencoder mini project.  I do not understand the solution, nor how to check it myself.  So this is 2 questions.

We start with 28*28 images.  These are fed through 3 convolutional layers, each with padding of 1, and each with a maxpooling to half the original dimensions.  What I don't understand is the last element?  Surely 2 rounds of maxpooling (28/2)/2 gives 7 and therefore a further maxpooling shouldn't be possible as it results in an odd number.  Can someone explain why this is the case to me?  The code to replicate is here:
'''

import torch
import numpy as np
from torchvision import datasets
import torchvision.transforms as transforms

# convert data to torch.FloatTensor
transform = transforms.ToTensor()

# load the training and test datasets
train_data = datasets.MNIST(root='data', train=True,
                                   download=True, transform=transform)
test_data = datasets.MNIST(root='data', train=False,
                                  download=True, transform=transform)

# Create training and test dataloaders
num_workers = 0
# how many samples per batch to load
batch_size = 20

# prepare data loaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, num_workers=num_workers)

import torch.nn as nn
import torch.nn.functional as F

# define the NN architecture
class ConvDenoiser(nn.Module):
    def __init__(self):
        super(ConvDenoiser, self).__init__()
        ## encoder layers ##
        # conv layer (depth from 1 --> 32), 3x3 kernels
        self.conv1 = nn.Conv2d(1, 32, 3, padding=1)
        # conv layer (depth from 32 --> 16), 3x3 kernels
        self.conv2 = nn.Conv2d(32, 16, 3, padding=1)
        # conv layer (depth from 16 --> 8), 3x3 kernels
        self.conv3 = nn.Conv2d(16, 8, 3, padding=1)
        # pooling layer to reduce x-y dims by two; kernel and stride of 2
        self.pool = nn.MaxPool2d(2, 2)

        ## decoder layers ##
        # transpose layer, a kernel of 2 and a stride of 2 will increase the spatial dims by 2
        self.t_conv1 = nn.ConvTranspose2d(8, 8, 3, stride=2)  # kernel_size=3 to get to a 7x7 image output
        # two more transpose layers with a kernel of 2
        self.t_conv2 = nn.ConvTranspose2d(8, 16, 2, stride=2)
        self.t_conv3 = nn.ConvTranspose2d(16, 32, 2, stride=2)
        # one, final, normal conv layer to decrease the depth
        self.conv_out = nn.Conv2d(32, 1, 3, padding=1)

    def forward(self, x):
        ## encode ##
        # add hidden layers with relu activation function
        # and maxpooling after
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        # add second hidden layer
        x = F.relu(self.conv2(x))
        x = self.pool(x)
        # add third hidden layer
        x = F.relu(self.conv3(x))
        x = self.pool(x)  # compressed representation

        ## decode ##
        # add transpose conv layers, with relu activation function
        x = F.relu(self.t_conv1(x))
        x = F.relu(self.t_conv2(x))
        x = F.relu(self.t_conv3(x))
        # transpose again, output should have a sigmoid applied
        x = F.sigmoid(self.conv_out(x))

        return x

# initialize the NN
model = ConvDenoiser()
print(model)

I wanted to try to understand this by passing a single image through the layers manually and see what the result was but this resulted in an error.  Can someone explain to me how I can see the shapes that pass through the layers?  Code is a bit messy but I left it there so you can see what I tried.

dataiter = iter(train_loader)
images, labels = dataiter.next()
# images = images.numpy()

# get one image from the batch
# img = np.squeeze(images[0])
img=images[0]

#create hidden layer
conv1 = nn.Conv2d(1, 32, 3, padding=1)

# z=torch.from_numpy(images[0])
z1=conv1(img)

Appreciate any insights you can give me.
Thanks,
J

Output:
Regarding your first question:
You can read in the documentation how the output shape of max-pooling is computed. You can max-pool odd-shaped tensors with even strides with or without padding. You need to be careful about the boundaries where some pixels may be lost.

Regarding your second question:
Your model expects a 4D input: batch-channel-height-width.
By selecting only one image from the batch (img=images[0]) you eliminate the batch dimension ending up with only a 3D tensor.
To fix this:

img=images[0:1, ...]  # select first image, but leave batch dimension as a singleton


Instruction: Bring any PyTorch cuda tensor in the range [0,1]

Input:
Suppose I have a PyTorch Cuda Float tensor x of the shape [b,c,h,w] taking on any arbitrary value allowed by Float Tensor range. I want to normalise it in the range [0,1].

I think of the following algorithm (but any other will also do).

Step1: Find minimum in each batch. Call it min and having shape [b,1,1,1].

Step2: Similarly find the maximum and call it max.

Step3: Use y = (x-min)/max. Alternatively use y = (x-min)/(max-min). I don't know which one will be better. y should have the same shape as that of x.

I am using PyTorch 1.3.1.

Specifically I am unable to get the desired min using torch.min(). Same goes for max.

I am going to use it for feeding it to pre-trained VGG for calculating perceptual loss (after the above normalisation i will additionally bring them to ImageNet mean and std). Due to some reason I cannot enforce [0,1] range during data loading part because the previous works in my area have a very specific normalisation algorithm which has to be used but some times does not ensures [0,1] bound but will be somewhere in its vicinity. That is why at the time computing perceptual loss I have to do this explicit normalisation as a precaution. All out of the box implementation of perceptual loss I am aware assume data is in [0,1] or [-1,1] range and so do not do this transformation.

Thankyou very much

Output:
Not the most elegant way, but you can do that using keepdim=True and specifying each of the dimensions:

channel_min = x.min(dim=1, keepdim=True)[0].min(dim=2,keepdim=True)[0].min(dim=3, keepdim=True)[0]
channel_max = x.max(dim=1, keepdim=True)[0].max(dim=2,keepdim=True)[0].max(dim=3, keepdim=True)[0]


Instruction: How to calculate perplexity for a language model using Pytorch

Input:
I'm fine-tuning GPT-2 model for language generation task using huggingface Transformers library-pytorch, and I need to calculate an evaluation score(perplexity) for the fine-tuned model. But I'm not sure how that can be done using loss. I would like to know how perplexity can be calculated for the model with sum_loss or mean loss or any other suggestions are also welcome. Any help is appriciated.

Edit:

outputs = model(article_tens, labels=article_tens)

        loss, prediction_scores = outputs[:2]
        loss.backward()
        sum_loss = sum_loss + loss.detach().data

given above is how I calculate loss for each batch of data for the fine-tuning task.

sum loss 1529.43408203125
loss 4.632936000823975
prediction_scores tensor([[[-11.2291,  -9.2614, -11.8575,  ..., -18.1927, -17.7286, -11.9215],
         [-67.2786, -63.5928, -70.7110,  ..., -75.3516, -73.8672, -67.6637],
         [-81.1397, -80.0295, -82.9357,  ..., -83.7913, -85.7201, -78.9877],
         ...,
         [-85.3213, -82.5135, -86.5459,  ..., -90.9160, -90.4393, -82.3141],
         [-44.2260, -43.1702, -49.2296,  ..., -58.9839, -55.2058, -42.3312],
         [-63.2842, -59.7334, -61.8444,  ..., -75.0798, -75.7507, -54.0160]]],
       device='cuda:0', grad_fn=<UnsafeViewBackward>)

The above given is when loss is printed for a batch only

Output:
As shown in Wikipedia - Perplexity of a probability model, the formula to calculate the perplexity of a probability model is:

The exponent is the cross-entropy. While logarithm base 2 (b = 2) is traditionally used in cross-entropy, deep learning frameworks such as PyTorch use the natural logarithm (b = e).

Therefore, to get the perplexity from the cross-entropy loss, you only need to apply torch.exp to the loss.

perplexity = torch.exp(loss)

The mean loss is used in this case (the 1 / N part of the exponent) and if you were to use the sum of the losses instead of the mean, the perplexity would get out of hand (exceedingly large), which can easily surpass the maximum floating point number, resulting in infinity.


Instruction: RMSE loss for multi output regression problem in PyTorch

Input:
I'm training a CNN architecture to solve a regression problem using PyTorch where my output is a tensor of 20 values. I planned to use RMSE as my loss function for the model and tried to use PyTorch's nn.MSELoss() and took the square root for it using torch.sqrt() for that but got confused after obtaining the results.I'll try my best to explain why. It's obvious that for a batch-size bs my output tensor's dimensions would be [bs , 20].I tried to implement and RMSE function of my own :

   def loss_function (predicted_x , target ):
        loss = torch.sum(torch.square(predicted_x - target) , axis= 1)/(predicted_x.size()[1]) #Taking the mean of all the squares by dividing it with the number of outputs i.e 20 in my case
        loss = torch.sqrt(loss)
        loss = torch.sum(loss)/predicted_x.size()[0]  #averaging out by batch-size
        return loss

But the output of my loss_function() and how PyTorch implements it with nn.MSELoss() differed . I'm not sure whether my implementation is wrong or am I using nn.MSELoss() in the wrong way.

Output:
The MSE loss is the mean of the squares of the errors. You're taking the square-root after computing the MSE, so there is no way to compare your loss function's output to that of the PyTorch nn.MSELoss() function — they're computing different values.

However, you could just use the nn.MSELoss() to create your own RMSE loss function as:

loss_fn = nn.MSELoss()
RMSE_loss = torch.sqrt(loss_fn(prediction, target))
RMSE_loss.backward()

Hope that helps.


Instruction: Do ReLU1 in PyTorch

Input:
I want to use ReLU1 non-linear activation. ReLU1 is linear in [0,1] but clamps values less than 0 to 0 and clamps values more than 1 to 1.

It will be used only for the last layer of my deep net in PyTorch having a really high definition output of 2048x4096. Since the code has to be highly optimized in terms of speed and memory I do not know which of the following will be the best implementation.

Following are the two implementations I can think of for the tensor x:-

x.clamp_(min=0.0, max=1.0)
For this I am unable to see the source code given in its docs. So do not know if its the best choice. I will prefer in place operation since backpropagation can happen through it.

The second alternative I have is to use torch.nn.functional.hardtanh_(x, min_val=0.0, max_val=1.0). This is definitely a in place function and the source code says that it uses the C++ file torch._C._nn.hardtanh(input, min_val, max_val) so I think it will be fast.

Please suggest which is the most efficient implementation and another one if possible.

Thankyou

Output: Without trying it, my guess is that clamp and hardtanh will have the same speed, and it will be hard to do this operation any faster if you optimize it in isolation. The arithmetic is trivial so this operation will be bottlenecked by GPU memory bandwidth. To run faster, you'd want to fuse this operation with the operation that produced x. If you don't want to write a custom kernel for the combined operation, you can try using TorchScript.


Instruction: How to change some value of a tensor into zero according to another tensor's value in pytorch?

Input:
I have two tensors: tensor a and tensor b. How can I change some value of tensor a according to the value of tensor b?

I know the codes following are right, but it runs pretty slow when the tensor is big. Is there any other method?

import torch
a = torch.rand(10).cuda()
b = torch.rand(10).cuda()
a[b > 0.5] = 0.

Output:
For this exact use case also consider

a * (b <= 0.5)

which seems to be the fastest out of the following

In [1]: import torch
   ...: a = torch.rand(3**10)
   ...: b = torch.rand(3**10)

In [2]: %timeit a[b > 0.5] = 0.
553 µs ± 17.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [3]: a = torch.rand(3**10)

In [4]: %timeit temp = torch.where(b > 0.5, torch.tensor(0.), a)
   ...:
49 µs ± 391 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [5]: a = torch.rand(3**10)

In [6]: %timeit temp = (a * (b <= 0.5))
44 µs ± 381 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [7]: %timeit a.masked_fill_(b > 0.5, 0.)
244 µs ± 3.48 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)


Instruction: Got `UnboundLocalError` error when measuring time for `torch.where`

Input:
When try to %timeit in Jupyter Notebook got error; Without it working fine.

  UnboundLocalError: local variable 'a' referenced before assignment

import torch
a = torch.rand(10)
b = torch.rand(10)

%timeit a = torch.where(b > 0.5, torch.tensor(0.), a)

What is happening here?

Output:
At first, I thought that's because %timeit only evaluates the time run by functions. But thanks to @Shiva who told me it can calculate the execution time of other things. And I checked the documentation here, and I found out this is true.

So, according to this answer, %timeit has a problem with re-assignment as the re-assignment to a causes the function to have an a local variable, hiding the global. In other words, you can use any other variable other that a to assign it to torch.where:

#this works
%timeit c = torch.where(b > 0.5, torch.tensor(0.), a) #c instead of a

# this works
%timeit torch.where(b > 0.5, torch.tensor(0.), a)

# this doesn't work
%timeit a = torch.where(b > 0.5, torch.tensor(0.), a)


Instruction: How to find c++ source code of torch.bmm of pytorch

Input:
I have trouble finding the source code of torch.bmm(), which is defined in https://pytorch.org/cppdocs/api/function_namespaceat_1aac51f71f807ca70fd210814114520c34.html#exhale-function-namespaceat-1aac51f71f807ca70fd210814114520c34.

I have confidence it is located in namespace at, since it is referenced as at::bmm in other places.
What I have searched through is :

The directory Aten https://github.com/pytorch/pytorch/tree/34877448216149024f44cbcab830169fdb2fa7fb/aten/src/ATen
The directory of caffe2 https://github.com/pytorch/pytorch/tree/74b65c32be68b15dc7c9e8bb62459efbfbde33d8/caffe2/core
direct search in github with keyword bmm of c++ file

but have found nothing. Is there any systematic way to locate a function(in this case, bmm) in such a large project ?

Output:
There is no (single) source for bmm per se. From ATen's Readme:

  ATen "native" functions are the modern mechanism for adding operators and functions to ATen (they are "native" in contrast to legacy functions, which are bound via TH/THC cwrap metadata). Native functions are declared in native_functions.yaml and have implementations defined in one of the cpp files in this directory.

bmm is declared in aten\src\ATen\native\native_functions.yaml:

- func: bmm(Tensor self, Tensor mat2) -> Tensor
  use_c10_dispatcher: full
  variants: function, method
  dispatch:
    CPU: bmm_cpu
    CUDA: bmm_cuda
    SparseCPU: bmm_sparse_cpu
    SparseCUDA: bmm_sparse_cuda
  supports_named_tensor: True

The implementations (like bmm_cpu) are to be found in aten\src\ATen\native\LinearAlgebra.cpp.


Instruction: Improving accuracy on a multi-class image classifiier

Input:
I am building a classifier using the Food-101 dataset. The dataset has predefined training and test sets, both labeled. It has a total of 101,000 images. I’m trying to build a classifier model with >=90% accuracy for top-1. I’m currently sitting at 75%. The training set was provided unclean. But now, I would like to know some of the ways I can improve my model and what are some of the things I’m doing wrong.

I’ve partitioned the train and test images into their respective folders. Here, I am using 0.2 of the training dataset to validate the learner by running 5 epochs.

np.random.seed(42)
data = ImageList.from_folder(path).split_by_rand_pct(valid_pct=0.2).label_from_re(pat=file_parse).transform(size=224).databunch()

top_1 = partial(top_k_accuracy, k=1)
learn = cnn_learner(data, models.resnet50, metrics=[accuracy, top_1], callback_fns=ShowGraph)
learn.fit_one_cycle(5)

epoch   train_loss  valid_loss  accuracy    top_k_accuracy  time
0       2.153797    1.710803    0.563498    0.563498    19:26
1       1.677590    1.388702    0.637096    0.637096    18:29
2       1.385577    1.227448    0.678746    0.678746    18:36
3       1.154080    1.141590    0.700924    0.700924    18:34
4       1.003366    1.124750    0.707063    0.707063    18:25

And here, I’m trying to find the learning rate. Pretty standard to how it was in the lectures:

learn.lr_find()
learn.recorder.plot(suggestion=True)

LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
Min numerical gradient: 1.32E-06
Min loss divided by 10: 6.31E-08

Using the learning rate of 1e-06 to run another 5 epochs. Saving it as stage-2

learn.fit_one_cycle(5, max_lr=slice(1.e-06))
learn.save('stage-2')

epoch   train_loss  valid_loss  accuracy    top_k_accuracy  time
0       0.940980    1.124032    0.705809    0.705809    18:18
1       0.989123    1.122873    0.706337    0.706337    18:24
2       0.963596    1.121615    0.706733    0.706733    18:38
3       0.975916    1.121084    0.707195    0.707195    18:27
4       0.978523    1.123260    0.706403    0.706403    17:04

Previously I ran 3 stages altogether but the model wasn’t improving beyond 0.706403 so I didn’t want to repeat it. Below is my confusion matrix. I apologize for the terrible resolution. Its the doing of Colab.

Since I’ve created an additional validation set, I decided to use the test set to validate the saved model of stage-2 to see how well it was performing:

path = '/content/food-101/images'
data_test = ImageList.from_folder(path).split_by_folder(train='train', valid='test').label_from_re(file_parse).transform(size=224).databunch()

learn.load('stage-2')
learn.validate(data_test.valid_dl)

This is the result:

[0.87199837, tensor(0.7584), tensor(0.7584)]

Output:
Try augmentations like RandomHorizontalFlip, RandomResizedCrop,
RandomRotate, Normalize etc from torchvision transforms. These always help a lot in classification problems.
Label smoothing and/or Mixup precision training.
Simply try using a more optimized architecture, like EfficientNet.
Instead of OneCycle, a longer, more manual training approach may help. Try Stochastic Gradient Descent with a weight decay of 5e-4 and a Nesterov Momentum of 0.9. Use Warmup Training of around 1-3 epochs, and then regular training of around 200 epochs. You could set a manual learning rate schedule or cosine annealing or some other scheme. This entire method will consume a lot more time and effort than the usual onecycle training, and should be explored only if other methods don't show considerable gains.


Instruction: Convert a Keras NN to a Pytorch NN

Input:
I'm a beginner with constructing neural networks in pytorch and Keras. I have the following Keras code for a variation of the AlexNet model:

def shallownet(nb_classes):
    global img_size
    model = Sequential()
    model.add(Conv2D(64, (5, 5), input_shape=img_size, data_format='channels_first'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same', data_format='channels_first'))

    model.add(Conv2D(64, (5, 5), padding='same', data_format='channels_first'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same', data_format='channels_first'))

    model.add(Flatten())
    model.add(Dense(384))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(192))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(nb_classes, activation='softmax'))
    return model

This is based on the C1/C3 model described on page 12 of this paper:
https://arxiv.org/pdf/1609.04836.pdf
And I want to convert this to the Pytorch version of the NN. Specifically I'm trying to get it in the form of:

class AlexNetOWT_BN(nn.Module):

    def __init__(self, num_classes=1000):
        super(AlexNetOWT_BN, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2,
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(64, 192, kernel_size=5, padding=2, bias=False),
            nn.BatchNorm2d(192),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(192, 384, kernel_size=3, padding=1, bias=False),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(384),
            nn.Conv2d(384, 256, kernel_size=3, padding=1, bias=False),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256),
            nn.Conv2d(256, 256, kernel_size=3, padding=1, bias=False),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256)
        )
        self.classifier = nn.Sequential(
            nn.Linear(256 * 6 * 6, 4096, bias=False),
            nn.BatchNorm1d(4096),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(4096, 4096, bias=False),
            nn.BatchNorm1d(4096),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(4096, num_classes)
        )

        #self.regime = {
        #    0: {'optimizer': 'SGD', 'lr': 1e-2,
        #        'weight_decay': 5e-4, 'momentum': 0.9},
        #    10: {'lr': 5e-3},
        #    15: {'lr': 1e-3, 'weight_decay': 0},
        #    20: {'lr': 5e-4},
        #    25: {'lr': 1e-4}
        #}
        self.regime = {
            0: {'optimizer': 'SGD', 'lr': 1e-2,
                'weight_decay': 5e-4, 'momentum': 0.9},
            20: {'lr': 1e-3},
            40: {'lr': 1e-4}
        }
        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])
        self.input_transform = {
            'train': transforms.Compose([
                transforms.Scale(256),
                transforms.RandomCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize
            ]),
            'eval': transforms.Compose([
                transforms.Scale(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                normalize
            ])
        }

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, 256 * 6 * 6)
        x = self.classifier(x)
        return x

The original AlexNet code for self.features is here (I switched the order of things above; not sure if that's allowed or not):

self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2,
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(64, 192, kernel_size=5, padding=2, bias=False),
            nn.BatchNorm2d(192),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(192, 384, kernel_size=3, padding=1, bias=False),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(384),
            nn.Conv2d(384, 256, kernel_size=3, padding=1, bias=False),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256),
            nn.Conv2d(256, 256, kernel_size=3, padding=1, bias=False),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256)
        )

I am specifically confused about how to change the features and the classifier. The description of the model I want is here:
https://arxiv.org/pdf/1609.04836.pdf on page 12, section B.3.

Thanks so much

Output:
Here's the code:

class AlexNet(nn.Module):

    def __init__(self, num_classes=10):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=5, padding=2,
                      bias=False),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Conv2d(64, 64, kernel_size=5, padding=2, bias=False),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
        )
        self.classifier = nn.Sequential(
            nn.Linear(64 * 7 * 7, 384, bias=False),
            nn.BatchNorm1d(384),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(384, 192, bias=False),
            nn.BatchNorm1d(192),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(192, num_classes)
        )
        self.regime = {
            0: {'optimizer': 'SGD', 'lr': 1e-3,
                'weight_decay': 5e-4, 'momentum': 0.9},
            60: {'lr': 1e-2},
            120: {'lr': 1e-3},
            180: {'lr': 1e-4}
        }

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, 64 * 7 * 7)
        x = self.classifier(x)
        return F.log_softmax(x)

def cifar10_shallow(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 10)
    return AlexNet(num_classes)

def cifar100_shallow(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 100)
    return AlexNet(num_classes)

Written by Wei Wen. smoothout repository


Instruction: How to transform a 2D and index tensors for torch.nn.utils.rnn.pack_sequence

Input:
I have a sequences collection in the following form:

sequences = torch.tensor([[2,1],[5,6],[3,0])
indexes = torch.tensor([1,0,1])

that is, the sequence 0 is made of just [5,6], and the sequence 1 is made of [2,1] , [3,0]. Mathematically sequence[i] = { sequences[j] such that i = indexes[j] }

I need to feed these sequences into an LSTM. Since these are variable-length sequences, pytorch documentation states to use something like torch.nn.utils.rnn.pack_sequence.

Sadly, this method and its like want, as input, a list of tensors where each of them is a L x *, with L being the length of the single sequence.

How can build something that can be fed into a pytorch LSTM?

P.s. throughout the code I work with these tensors using scatter and gather functionalities but I can't find a way to use them to achieve this goal.

Output:
I found an alternative and more efficient way to separate the sequences:

sequences = torch.tensor([[2,1],[5,6],[3,0]])
indexes = torch.tensor([1,0,1])

sorted_src = src[indexes.argsort()]
indexes_count = torch.unique(indexes, return_counts=True)[1]

splitted = torch.split(sorted_src, indexes_count.tolist(), dim=0)

This method is almost 3 times faster then the one proposed by @Mercury.

  Measured using timeit module with sequences being a (5000,256) tensor and indexes being (1500)


Instruction: How to fix documentation format in VS Code for python kite?

Input: I am using pytorch in VSCode but the format of the documentation is just terrible (see image). Do you have any pointers on how we can set the documentation format in VSCode? I believe that documentation contains LaTeX formulas.... Check the image in the link

Output: There's currently nothing you can do. It contains LaTeX and the extension does not render them as such.


Instruction: In PyTorch, how to convert the cuda() related codes into CPU version?

Input:
I have some existing PyTorch codes with cuda() as below, while net is a MainModel.KitModel object:
net = torch.load(model_path)
net.cuda()

and
im = cv2.imread(image_path)
im = Variable(torch.from_numpy(im).unsqueeze(0).float().cuda())

I want to test the code in a machine without any GPU, so I want to convert the cuda-code into CPU version. I tried to look at some relevant posts regarding the CPU/GPU switch of PyTorch, but they are related to the usage of device and thus doesn't apply to my case.

Output:
As pointed out by kHarshit in his comment, you can simply replace .cuda() call with .cpu():
net.cpu()
# ...
im = torch.from_numpy(im).unsqueeze(0).float().cpu()

However, this requires changing the code in multiple places every time you want to move from GPU to CPU and vice versa.
To alleviate this difficulty, pytorch has a more "general" method .to().
You may have a device variable defining where you want pytorch to run, this device can also be the CPU (!).
for instance:
if torch.cuda.is_available():
  device = torch.device("cuda")
else:
  device = torch.device("cpu")

Once you determined once in your code where you want/can run, simply use .to() to send your model/variables there:
net.to(device)
# ...
im = torch.from_numpy(im).unsqueeze(0).float().to(device)

BTW,
You can use .to() to control the data type (.float()) as well:
im = torch.from_numpy(im).unsqueeze(0).to(device=device, dtype=torch.float)

PS,
Note that the Variable API has been deprecated and is no longer required.


Instruction: How is Vocab and Integer (one hot) representation stored and what does the ('string', int) tuple means in torchtext.vocab()?

Input:
I am trying to train an RNN for binary classification. I have my vocab made from 1000000 words and please find the below outputs...

text_field = torchtext.data.Field(tokenize=word_tokenize)

print(text_field.vocab.freqs.most_common(15))
>>
[('.', 516822), (',', 490533), ('the', 464796), ('to', 298670), ("''", 264416), ('of', 226307), ('I', 224927), ('and', 215722), ('a', 211773), ('is', 180965), ('you', 180359), ('``', 165889), ('that', 156425), ('in', 138038), (':', 132294)]

print(text_field.vocab.itos[:15])
>>
['<unk>', '<pad>', '.', ',', 'the', 'to', "''", 'of', 'I', 'and', 'a', 'is', 'you', '``', 'that']

text_field.vocab.stoi
>>
{'<unk>': 0,'<pad>': 1,'.': 2,',': 3,'the': 4,'to': 5,"''": 6,'of': 7,'I': 8,'and': 9,'a': 10, 'is': 11,'you': 12,'``': 13,'that': 14,'in': 15,....................

The documentation says:

freqs – A collections.Counter object holding the frequencies of tokens in the data used to build the Vocab.
stoi – A collections.defaultdict instance mapping token strings to numerical identifiers.
itos – A list of token strings indexed by their numerical identifiers.

Which not comprehensible by me.

Can Some one Please explain what are these by giving the intuition of each of those?

For example, if the is represented by 4, then does it mean that if a sentence contains the word the,

Is it going to be a 1 at location 4?  OR
Is it going to be a 1 at position 464796 OR
Is it going to be a 4 at position 464796??

What happens when multiple the are there there??

Output:
If "the" is represented by 4, then that means that

itos[4] is "the"
stoi["the"] is 4
there is a tuple ('the', <count>) somewhere in freqs, where count is the number of times that 'the' appears in your input text. That count has nothing to do with its numerical identifier 4.


Instruction: In pytorch, what situations the loss function need to inherit nn.module?

Input: I am confused about the loss function in PyTorch. Some people define the loss function as a normal python function while others define the loss function by defining a class that inherits nn.Module. So I want to know what situations we need to define the loss function by inheriting nn.Module? Many thanks.

Output:
Generally, inheritance from nn.Module is only necessary when you want to have trainable variables in this module, otherwise it's optional to inherit it.

So same applies to loss functions, if it contains no such variables (which I assume is the major case), no inheritance is needed.


Instruction: Batchnorm2d Pytorch - Why pass number of channels to batchnorm?

Input: Why do I need to pass the previous nummber of channels to the batchnorm? The batchnorm should normalize over each datapoint in the batch, why does it need to have the number of channels then ?

Output:
Batch normalisation has learnable parameters, because it includes an affine transformation.

From the documentation of nn.BatchNorm2d:

  The mean and standard-deviation are calculated per-dimension over the mini-batches and γ and β are learnable parameter vectors of size C (where C is the input size). By default, the elements of γ are set to 1 and the elements of β are set to 0.

Since the norm is calculated per channel, the parameters γ and β are vectors of size num_channels (one element per channel), which results in an individual scale and shift per channel. As with any other learnable parameter in PyTorch, they need to be created with a fixed size, hence you need to specify the number of channels

batch_norm = nn.BatchNorm2d(10)

# γ
batch_norm.weight.size()
# => torch.Size([10])

# β
batch_norm.bias.size()
# => torch.Size([10])

Note: Setting affine=False does not use any parameters and the number of channels wouldn't be needed, but they are still required, in order to have a consistent interface.