Instruction: Why doesn't nn.Sequential contain a softmax output layer in the example?

Input:
The example from PyTorch's official tutorial has the following ConvNet. My understanding is that the output layer uses a softmax to estimate the digit an image corresponds to. Why doesnt the code have a softmax layer or fully connected layer?

model = nn.Sequential(
    nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d(1),
    Lambda(lambda x: x.view(x.size(0), -1)),
)

opt = optim.SGD(model.parameters(), lr=lr, momentum=0.9)

Output:
This is a very good question! The reason why no fully-connected layer is used is because of a technique called Global Average Pooling, implemented via nn.AdaptiveAvgPool2d(1). The benefits of this operation over fc layers were introduced in this paper, including reducing the number of model parameters while preserving performance, acting as a regulariser, and modelling deep localisation information. GAP can be used in place of fc, as well as before a subsequent fc layer.

As for why there is no softmax layer, I think that this is because they use the CrossEntropyLoss loss function in the backend. This function takes in raw logits and combines nn.LogSoftmax() and nn.NLLLoss() in one computation. So there is no need to perform an additional softmax function before loss evaluation.


Instruction: Creating a stack of convolutional layers using for loop in forward function of a pytoch class for Residual block

Input:
I'm defining a residual block in pytorch for ResNet in which you can input how many convolutional layers you want to have and not necessarily two. This is done through a parameter named nc (number of Convs). The first layer gets ni as the number of input nf number of filters. But from second layer on I put them in a for loop. Here's my code:

class ResBlock(nn.Module):
    def __init__(self, ni, nf,nc=2):
        super().__init__()
        self.conv1 = nn.Conv2d(ni,nf, kernel_size=3, stride=2, padding=0)
        self.conv2 = nn.Conv2d(nf,nf, kernel_size=3, stride=1, padding=0)
        self.conv1x1 = nn.Conv2d(ni, nf, kernel_size=1, stride=1, padding=0)
        self.nc = nc
    def forward(self, x):
        y = self.conv1(x)
        for i in range(self.nc-1):
            y = self.conv2(y)
            print(torch.mean(y))
        return self.conv1x1(x) + y

But no matter what value I give to nc, it always returns 2 convs with kernel size 3. I'm not sure if for loop can really do this job in pytorch but it was working when I used functional API in Keras. Could anyone help me understand what's going on?

Output:
Yeah, printing a nn.Module object is often misleading. When you print, you get:

# for ni=3, nf=16
ResBlock(
  (conv1): Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2))
  (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1))
  (conv1x1): Conv2d(3, 16, kernel_size=(1, 1), stride=(1, 1))
)

because these are the only 3 Modules you registered in the __init__ of the ResBlock.

The actual forward can (and in your case will) be doing something completely different.


Instruction: Predicting the next track using a vanilla RNN in PyTorch

Input:
For some context, I have a set of 37 playlists of 12 tracks long. Each track has been hand-selected in a certain way. Early songs in the playlist are generally more chilled and as the playlist progresses tracks begin to increase in tempo. I decided to commit to a project and build a deep playlist generator.

I am implementing a many-to-many vanilla RNN in PyTorch and am seeking clarity on how to train the RNN one playlist at a time, where each track is then parsed and the model predicts the features of the next track.

Pictured is a Many-to-many RNN - for this case - each red box is the current track's features and the opposite blue box is the predicted next track's features:

The feature set (9), X, looks like so:

The target y simply mirrors the above feature set of the next track.

For my RNN Class it looks like so:

class RNNEstimator(nn.Module):
    def __init__(self, input_size=9, hidden_size=30, output_size=9):
        super(RNNEstimator, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)

    def forward(self, inp, hidden):
        print("inp", inp.shape)
        print("hid", hidden.shape)
        combined = torch.cat((inp, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

This is taken from the PyTorch tutorials page. However, I have adapted the RNN Class to output 9 features rather than a binary classification.

The playlist dataset has been processed into a tensor of shape torch.Size([37, 12, 18]), and stride (12, 1, 444)) - meaning 37 playlist, 12 tracks longs with 9 X_features + 9 y_features (18).

The train_rnn function:

# Model Initiation
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = RNNEstimator(9, 30, 9)
optimizer = optim.Adam(model.parameters(), lr=0.001)
loss_fn = torch.nn.L1Loss()

# Training function for RNN
def train_rnn(model, train_loader, epochs, criterion, optimizer, device):

    model.train() # Make sure that the model is in training mode.

    # training loop is provided
    for epoch in range(1, epochs + 1):

        for batch in train_loader:

            total_loss = 0

            # get data
            batch_x = batch[:, :9, :].float().squeeze()
            batch_y = batch[:, 9:, :].float()

            batch_x = batch_x.to(device)
            batch_y = batch_y.to(device)

            optimizer.zero_grad()

            hidden = model.initHidden()

            # For each track in batch/playlist
            # TODO: THIS NEEDS WORK
            for x, y in zip(batch_x, batch_y):
                output, hidden = model(x, hidden)
                loss = criterion(output, y)
                loss.backward()
                optimizer.step()
                total_loss += loss.data.item()

        if epoch % 10 == 0:
            print('Epoch: {}/{}.............'.format(epoch, epochs), end=' ')
            print("Loss: {:.4f}".format(loss.item()))

What I am trying to understand is how to train this model one playlist per batch?

I am getting an error from the cat function like so:

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

The model should parse each track (t) - via the forward method - then output the next track (t+1). The hidden state will reset each playlist given they are independent of one another.

Output:
Even though you want to run with a batch size of 1, your input (x) still needs a batch dimension. Try:

output, hidden = model(x.unsqueeze(0), hidden)


Instruction: what is the difference between if-else statement and torch.where in pytorch?

Input:
See the code snippet:

import torch
x = torch.tensor([-1.], requires_grad=True)
y = torch.where(x > 0., x, torch.tensor([2.], requires_grad=True))
y.backward()
print(x.grad)

The output is tensor([0.]), but

import torch
x = torch.tensor([-1.], requires_grad=True)
if x > 0.:
    y = x
else:
    y = torch.tensor([2.], requires_grad=True)
y.backward()
print(x.grad)

The output is None.

I'm confused that why the output of torch.where is tensor([0.])?

update

import torch
a = torch.tensor([[1,2.], [3., 4]])
b = torch.tensor([-1., -1], requires_grad=True)
a[:,0] = b

(a[0, 0] * a[0, 1]).backward()
print(b.grad)

The output is tensor([2., 0.]). (a[0, 0] * a[0, 1]) is not in any way related to b[1], but the gradient of b[1] is 0 not None.

Output:
Tracking based AD, like pytorch, works by tracking. You can't track through things that are not function calls intercepted by the library.  By using an if statement like this, there's no connection between x and y, whereas with where, x and y are linked in the expression tree.

Now, for the differences:

In the first snippet, 0 is the correct derivative of the function x ↦ x > 0 ? x : 2 at the point -1 (since the negative side is constant).
In the second snippet, as I said, x is not in any way related to y (in the else branch).  Therefore, the derivative of y given x is undefined, which is represented as None.

(You can do such things even in Python, but that requires more sophisticated technology like source transformation.  I don't thing it is possible with pytorch.)


Instruction: How to add pooling layer to BERT QA for large text

Input:
I'm trying to implement a Question answering system that deal with large input text: so the idea is to split the large input text into subsequences of 510 tokens, after I will generate the representation of each sequence independently and using a pooling layer to generate the final representation of the input sequence.

I using the CamemBERT model for French language.

I have tried the following code:

class CamemBERTQA(nn.Module):

# the initialization of the model
   def __init__(self, do_lower_case: bool = True):
       super(CamemBERTQA, self).__init__()
       self.config_keys = ['do_lower_case']
       self.do_lower_case = do_lower_case
       self.camembert = CamembertForQuestionAnswering.from_pretrained('fmikaelian/camembert-base-fquad')
       self.tokenizer = CamembertTokenizer.from_pretrained('fmikaelian/camembert-base-fquad', do_lower_case=do_lower_case)
       self.cls_token_id = self.tokenizer.convert_tokens_to_ids([self.tokenizer.cls_token])[0]
       self.sep_token_id = self.tokenizer.convert_tokens_to_ids([self.tokenizer.sep_token])[0]
       self.pool = nn.MaxPool2d(2, 2)

# Split long input text into subsequences with overlapping
   def split_text(self, text, max_length, overlapp): #511 max
       f = []
       text = text.split()
       for i in range(0, int(len(text)-overlapp),(max_length-overlapp)):
           f.append(" ".join(text[i:i+max_length]))
#             print (f)
       return f

# Generate representation of a text,
   def text_representation(self, l): #  l here is a list
       result = []
       for i in l:
           input_ids = torch.tensor([self.tokenizer.encode(i, add_special_tokens=True)])
           with torch.no_grad():
               last_hidden_states = model(input_ids)[0]  # Models outputs are now tuples
               result.append(last_hidden_states)
#                     print(last_hidden_states[0])
       return result

   def forward(self, text, input_ids):
       # Split input text to subsequences of 511 with overlapping
       subsequences = self.split_text(text, 511, 10)

       # Generate IDs of each subsequence (Sequence representation)
       input_ids_list = self.text_representation(subsequences)
       print("input_ids_list")

       # Pooling layer
#         pool = self.pool(...)

###########      The problem is here: how can I add a pooling layer                  #################

#         input_ids = # the final output of the pooling layer, the result should contain 510 elements/tokens

       # generate the start and end logits of the answer
       start_scores, end_scores = self.camembert(torch.tensor([input_ids]))
       start_logits = torch.argmax(start_scores)
       end_logits = torch.argmax(end_scores)+1
       outputs = (start_logits, end_logits,)
#         print(outputs)

       return outputs

Since I'm a beginner with pyTorch, I'm not sure about if the code should be like that.

Please if you have any advice or if you need any further information contact me.

Output:
I'm pretty new to all of this myself, but maybe this could help you:
    def max_pooling(input_tensor, max_sequence_length):

        mxp = nn.MaxPool2d((max_sequence_length, 1),stride=1)
        return mxp(input_tensor)


Instruction: What does "conda install pytorch torchvision cudatoolkit=10.2 -c pytorch" install?

Input:
I tried installing PyTorch on my system with not just the pip install pytorch -c pytorch command but with conda install pytorch torchvision cudatoolkit=10.2 -c pytorch but I see a very long command prompt running since last 2 hours giving a very large outputs.
Is the process going good? I've CUDA 10.2 installed and also tensorflow-gpu and cuDNN too.
I see these arguments being shown.
Here's the long list of commands running since past 2 hours

https://drive.google.com/file/d/1D4p9bfxNHXZfe8PCjc45OPlPZfasxk_4/view?usp=sharing

Thanks in advance.

Output:
The preferred way of installing PyTorch is through Anaconda, it has some of the common dependencies (packages) pre-installed and saves you a lot of time. Try a clean install of Conda and run:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

The main difference between Anaconda and a vanilla Python installation would be the packages that come pre-installed and the source of those packages. Conda has it's own Python environment, own set of packages and Conda CLI (and a GUI now) to manage the environment.
The conda command can be thought of as pip, but the difference being that conda can install libraries and packages that are not only for Python.

Refer to here for more details: https://www.anaconda.com/understanding-conda-and-pip/

As for your log, I don't see anything wrong with it. Just let it do it's job.
If there is an error message I missed on there, let me know and I'll take a look.


Instruction: how does BatchNorm1d() method whithin the torch library work?

Input: I'm learning pytorch, I don;t know if this question is stupid but I can't find the official web for explaining nn.batchnorm1d. I'm wondering how torch.nn.BatchNorm1d(d1) work? I know that batch norm is about making mean and variance of a batch of example to be 0 and 1 respectively. I'm wondering if there is nn.batchnorm2d, if so, what does it do? what is the d1 parameter ?

Output:
BatchNorm1d normalises data to 0 mean and unit variance for 2/3-dimensional data (N, C) or (N, C, L), computed over the channel dimension at each (N, L) or (N,) slice; while BatchNorm2d does the same thing for 4 dimensions (N, C, H, W), computed over the channel dimension at each (N, H, W) slice.

Which one to use depends on the dimensionality of input data. For instance in image processing, feature maps ususally have 2 spatial dimensions (N, C, H, W), so BatchNorm2d is useful here. However for some NLP tasks, if there is only the length dimension to consider, one would use BatchNorm1d. For both functions, the d1 parameter is the number of features, and equals dim C of the input tensor.


Instruction: Wrong Number of Init Arguments for Tanh in Pytorch

Input:
For a homework assignment, I am implementing a simple neural network in Python using Pytorch. Here is my network class:

class Net(torch.nn.Module):
    def __init__(self, layer_dims, activation="sigmoid"):
        super(Net, self).__init__()
        layers = []
        if activation == 'sigmoid':
            for i in range(1, len(layer_dims) - 1):
                layers.append(nn.Sigmoid(layer_dims[i - 1], layer_dims[i]))
                layers.append(nn.Sigmoid(layer_dims[i - 1]))
            layers.append(nn.Sigmoid(layer_dims[-2], layer_dims[-1]))
            layers.append(nn.Sigmoid())
        elif activation == 'relu':
            for i in range(1, len(layer_dims) - 1):
                layers.append(nn.ReLu(layer_dims[i - 1], layer_dims[i]))
                layers.append(nn.ReLU(layer_dims[i - 1]))
            layers.append(nn.ReLu(layer_dims[-2], layer_dims[-1]))
            layers.append(nn.ReLu())
        elif activation == 'tanh':
            for i in range(1, len(layer_dims) - 1):
                layers.append(nn.Tanh(layer_dims[i - 1], layer_dims[i]))
                layers.append(nn.Tanh(layer_dims[i - 1]))
            layers.append(nn.Tanh(layer_dims[-2], layer_dims[-1]))
            layers.append(nn.Tanh())
        elif activation == 'identity':
            for i in range(1, len(layer_dims) - 1):
                layers.append(nn.Identity(layer_dims[i - 1], layer_dims[i]))
                layers.append(nn.Identity(layer_dims[i - 1]))
            layers.append(nn.Identity(layer_dims[-2], layer_dims[-1]))
            layers.append(nn.Identity())

        self.out = nn.Sequential(*layers)

    def forward(self, input):
        return self.out(input)

def train(data, labels, n, l, activation='sigmoid'):
    if activation not in ['sigmoid','identity','tanh','relu']:
        return
    net = Net([l for i in range(0,n)], activation)
    optim = torch.optim.Adam(net.parameters())
    for i in range(0,5):
        ypred = net.forward(torch.Tensor(data))
        ypred.backward()
        optim.step()
        optim.zero_grad()
    ypred = net.forward(torch.Tensor(data))
    return (net, torch.nn.CrossEntropyLoss(ypred, labels), net.parameters(), ypred)

When testing this, I have been trying to run the following code segment:

for i in range(3,5):
    for num in [10,30,50]:
        print(train(data.get('X_trn'), data.get('y_trn'), i, num, activation='tanh'))

Which is erroring out with a TypeError, saying that init() takes 1 positional argument when 3 is given.

<ipython-input-30-376b6c739a71> in __init__(self, layer_dims, activation)
     18         elif activation == 'tanh':
     19             for i in range(1, len(layer_dims) - 1):
---> 20                 layers.append(nn.Tanh(layer_dims[i - 1], layer_dims[i]))
     21                 layers.append(nn.Tanh(layer_dims[i - 1]))
     22             layers.append(nn.Tanh(layer_dims[-2], layer_dims[-1]))

TypeError: __init__() takes 1 positional argument but 3 were given

This error has appeared when I switch the activation function as well. I am unsure what the issue is, because as far as I know, when you create a layer you need to give the input and output dimensions, which is what I have. Any help sorting this out would be appreciated.

Output:
The error clearly says, Tanh only takes 1 argument, a tensor.

From documentation, https://pytorch.org/docs/stable/nn.html

Tanh

class
torch.nn.Tanh
[source]

    Applies the element-wise function:
    Tanh(x)=tanh(x)=ex−e−xex+e−x\text{Tanh}(x) = \tanh(x) = \frac{e^x - e^{-x}} {e^x + e^{-x}} Tanh(x)=tanh(x)=ex+e−xex−e−x

    Shape:

            Input: (N,∗)(N, *)(N,∗) where * means, any number of additional dimensions

            Output: (N,∗)(N, *)(N,∗) , same shape as the input

You have so many mistakes it's hard to fix them all, you also didn't give a data sample.

Activation functions accept a single tensor, you are passing two random list elements. Usually, you can use torch.cat to concatenate two tensors. I would suggest you start with a simpler model and read the documentation first.


Instruction: Can't train ResNet using gpu with pytorch

Input:
I'm trying to use gpu to train a ResNet architecture on CIFAR10 dataset. Here's my code for ResNet :

import torch
import torch.nn as nn
import torch.nn.functional as F

class ResNetBlock(nn.Module):

    def __init__(self, in_planes, planes, stride=1):
        super(ResNetBlock, self).__init__()
        self.stride = stride
        self.in_planes=in_planes
        self.planes = planes
        if stride!=1:

          self.fx = nn.Sequential(nn.Conv2d(in_planes, planes, 3, stride=2,
                                            padding=1),
                                  nn.ReLU(),
                                  nn.Conv2d(planes, planes,3, padding=1))

        else:
          self.fx = nn.Sequential(nn.Conv2d(planes, planes, 3, padding = 1),
                                  nn.ReLU(),
                                  nn.Conv2d(planes, planes,3, padding=1))

    def forward(self, x):

      if self.stride ==1:
        fx = self.fx(x)
        id = nn.Sequential()
        out = fx + id(x)
        relu = nn.ReLU()
        return relu(out)

      else:

        fx = self.fx(x)
        id = nn.Conv2d(self.in_planes, self.planes, 2, stride = 2)
        out = fx + id(x)
        relu = nn.ReLU()
        return relu(out)

class ResNet(nn.Module):

  def __init__(self, block, num_blocks, num_classes=10, num_filters=16, input_dim=3):
      super(ResNet, self).__init__()
      self.in_planes = num_filters

      self.conv1 = nn.Conv2d(input_dim, num_filters, kernel_size=3, stride=1, padding=1, bias=False)
      self.bn1 = nn.BatchNorm2d(num_filters)

      layers = []
      plane = num_filters

      for nb in num_blocks:

        layer = self._make_layer(block,plane ,nb,2)
        layers.append(layer)
        plane*=2

      self.layers = nn.Sequential(*layers)

      self.linear = nn.Linear(2304, num_classes)

  def _make_layer(self, block, planes, num_blocks, stride):
      layers = []
      block1 = ResNetBlock(planes, 2*planes, stride = 2)
      planes *=2
      layers.append(block1)

      for i in range(1,num_blocks):
        block = ResNetBlock(planes, planes, stride =1)
        layers.append(block)

      return nn.Sequential(*layers)

  def forward(self, x):
      out = F.relu(self.bn1(self.conv1(x)))
      out = self.layers(out)

      out = F.avg_pool2d(out, 4)
      out = out.view(out.size(0), -1)
      out = self.linear(out)
      return out

# (1 + 2*(1 + 1) + 2*(1 + 1) + 2*(1 + 1) + 2*(1 + 1)) + 1 = 18
def ResNet18():
    return ResNet(ResNetBlock, [2,2,2,2])

Then I train the network using gpu :

net = ResNet18()
net = net.to('cuda')
train2(net, torch.optim.Adam(net.parameters(), lr=0.001), trainloader, criterion, n_ep=3)

And I get the error :

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

which is annoying because my weights should be cuda as well because of the resnet.cuda().

With another network the train function works well, so it must come from the classes mentioned above.

Also, next(resnet.parameters()).is_cuda returns True.

Update : Here's my training function.

def train(net, optimizer, trainload, criterion, n_ep=10, cuda = True):
  if cuda:
    net = net.to('cuda')

  for epoch in range(n_ep):
    for data in trainload:

      inputs, labels = data
      if cuda:
        inputs = inputs.type(torch.cuda.FloatTensor)
        labels = labels.type(torch.cuda.LongTensor)

      optimizer.zero_grad()

      print(next(net.parameters()).is_cuda)
      ## this actually prints "True" !

      outputs = net.forward(inputs)
      loss = criterion(outputs, labels)
      loss.backward()
      optimizer.step()

  return net

The thing is, this training function works well with another type of net. For example is used this one (AlexNet) :

class AlexNet(nn.Module):

    def __init__(self, num_classes=1000):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(nn.Conv2d(3,64,11), nn.ReLU(),nn.MaxPool2d(2, stride = 2), nn.Conv2d(64,192,5),
                                     nn.ReLU(), nn.MaxPool2d(2, stride = 2), nn.Conv2d(192,384,3),
                                     nn.ReLU(),nn.Conv2d(384,256,3), nn.ReLU(), nn.Conv2d(256,256,3), nn.ReLU())
        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 6 * 6, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),)

    def forward(self, x):
        x = self.features(x)
        x = self.avgpool(x)
        x = x.view(x.size(0), 256 * 6 * 6)
        x = self.classifier(x)
        return x

and with this one the gpu training works well.

There's something else I don't understand. I tried to train a network that I moved to GPU (using .cuda()  ) with training data that I did not move to GPU (on purpose). And this time I get the error that weights type is torch.cuda and data type isn't.

EDIT : I thought it had to do with using nn.ModuleList instead of regular python lists. However I tried that and it has not fixed the issue.

Output:
We would need a snippet of your training loop to better determine your error.

I am asuming that somewhere on that loop you have some lines of code which do the following:

for data, label in CifarDataLoader:
     data, label = data.to('cuda'), label.to('cuda')

My first guess would be to add a line just before the for loop ->

resnet = resnet.to('cuda')

Let me know if this works, if not I would need more of your code to find the error.


Instruction: ValueError: Target and input must have the same number of elements. target nelement (50) != input nelement (100)

Input:
I'm new to Pytorch so I tried to learn it by creating simple dogs vs cats classification.
The code:

class DogCatClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.conv3 = nn.Conv2d(64, 128, 5)

        self.fc1 = nn.Linear(512, 256)
        self.fc2 = nn.Linear(256, 2)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        print("1-st: ", x.shape)

        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        print("2-nd: ", x.shape)

        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))
        print("3-rd: ", x.shape)

        x = torch.flatten(x, start_dim=1)

        x = F.relu(self.fc1(x))
        print("6-th: ", x.shape)

        x = self.fc2(x)  # bc this is our output layer. No activation here.
        print("7-th: ", x.shape)

        x = F.sigmoid(x)
        print("8-th: ", x.shape)

        return x

I pass a single batch of data (data shape is (50, 1, 50, 50)

model = DogCatClassifier()

images, labels = next(iter(train_loader))

preds = model(images)
print(pred)

loss = F.binary_cross_entropy(preds, labels)

My prediction shape is (50, 2), so as I understand F.binary_cross_entropy(preds, labels) checks both predictions from a single image and that's why I get 100 predictions against 50 labels. Coming from tensorflow I thought that I could just implement the same logic like using sigmoid as last activation and binary_cross_entropy as loss function. What I don't understand is how to make this piece of code work.

Output:
Your problem arises because you are using binary cross entropy instead of regular cross entropy. As the name implies, it checks weather the label is correct or not thus the shape of both tensors (preds and labels in your code) should be the same. As you are giving the confidence of both classes, the BCE loss function gets confused and the code crashes. You can either do two things:

1- Change to F.cross_entropy(preds, label) as your loss function.

2- Change your code to pick the maximum value as the target.

  pred = pred.argmax(dim=1, keepdim=True) # gets the max value

Let me know if this works, if it doesn't please update with the new error.


Instruction: How can I load a partial pretrained pytorch model?

Input:
I'm trying to get a pytorch model running on a sentence classification task. As I am working with medical notes I am using ClinicalBert (https://github.com/kexinhuang12345/clinicalBERT) and would like to use its pre-trained weights. Unfortunately the ClinicalBert model only classifies text into 1 binary label while I have 281 binary labels. I am therefore trying to implement this code https://github.com/kaushaltrivedi/bert-toxic-comments-multilabel/blob/master/toxic-bert-multilabel-classification.ipynb where the end classifier after bert is 281 long.

How can I load the pre-trained Bert weights from the ClinicalBert model without loading the classification weights?

Naively trying to load the weights from the pretrained ClinicalBert weights I get the following error:

size mismatch for classifier.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([281, 768]).
size mismatch for classifier.bias: copying a param with shape torch.Size([2]) from checkpoint, the shape in current model is torch.Size([281]).

I currently tried to replace the from_pretrained function from the pytorch_pretrained_bert package and pop the classifier weights and biases like this:

def from_pretrained(cls, pretrained_model_name, state_dict=None, cache_dir=None, *inputs, **kwargs):
    ...
    if state_dict is None:
        weights_path = os.path.join(serialization_dir, WEIGHTS_NAME)
        state_dict = torch.load(weights_path, map_location='cpu')
    state_dict.pop('classifier.weight')
    state_dict.pop('classifier.bias')
    old_keys = []
    new_keys = []
    ...

And I get the following error message:
INFO - modeling_diagnosis -   Weights of BertForMultiLabelSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']

In the end I would like to load the bert embeddings from the clinicalBert pretrained weights and have the top classifier weights initialized randomly.

Output: Removing the keys in the state dict before loading is a good start. Assuming you're using nn.Module.load_state_dict to load the pretrained weights then you'll also need to set the strict=False argument to avoid errors from unexpected or missing keys. This will ignore entries in the state_dict that aren't present in the model (unexpected keys) and, more importantly for you, will leave the missing entries with their default initialization (missing keys). For safety you can check the return value of the method to verify the weights in question are part of the missing keys and that there aren't any unexpected keys.


Instruction: Loss Function in Multi-GPUs training (PyTorch)

Input:
I use Pytorch and BERT to traing a model. Everithing works great on one GPU, but when I try to use multi GPUs I am getting an error:

ValueError                                Traceback (most recent call last)
<ipython-input-168-507223f9879c> in <module>()
     92         # single value; the `.item()` function just returns the Python value
     93         # from the tensor.
---> 94         total_loss += loss.item()
     95
     96         # Perform a backward pass to calculate the gradients.

ValueError: only one element tensors can be converted to Python scalars

Can someone help me what I am missing and how I should fix it?

Here is my code for training:

import random
seed_val = 42
random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)
loss_values = []
for epoch_i in range(0, epochs):

    t0 = time.time()
    total_loss = 0
    for step, batch in enumerate(train_dataloader):
        if step % 40 == 0 and not step == 0:
            elapsed = format_time(time.time() - t0)

        b_input_ids = batch[0].to(device).long()
        b_input_mask = batch[1].to(device).long()
        b_labels = batch[2].to(device).long()
        model.zero_grad()
        outputs = model(b_input_ids,
                    token_type_ids=None,
                    attention_mask=b_input_mask,
                    labels=b_labels)

        loss = outputs[0]
        total_loss += loss.item()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        scheduler.step()
    avg_train_loss = total_loss / len(train_dataloader)
    loss_values.append(avg_train_loss)
    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(format_time(time.time() - t0)))

And here is my code for the model:

from transformers import BertForSequenceClassification, AdamW, BertConfig
model_to_parallel = BertForSequenceClassification.from_pretrained(
    "./bert_cache.zip",
    num_labels = 2,
    output_attentions = False,
    output_hidden_states = False,
)
model = nn.DataParallel(model_to_parallel,  device_ids=[0,1,2,3])
model.to(device)

Output:
After loss loss = outputs[0] the loss is a multi-element tensor, the size is number of GPUs.

You can use loss = outputs[0].mean() instead.


Instruction: PyTorch equivalent of numpy's np.random.RandomState

Input:
I'm looking for a way to create random objects without actually altering the pytorch global seed.

i.e. an equivalent to numpy's:

rand_gen = np.random.RandomState(seed)
rand_gen.randint(0, 256, self.image_dim))  # for example

Output:
You could pass your torch.Generator manually to the random function.
 I think this code should work:

gen0 = torch.Generator()
gen1 = torch.Generator()

gen0 = gen0.manual_seed(0)
gen1 = gen1.manual_seed(1)
torch.rand(5, generator=gen0)
torch.rand(5, generator=gen0)
torch.rand(5, generator=gen1)
torch.rand(5, generator=gen1)

gen0 = gen0.manual_seed(0)
gen1 = gen1.manual_seed(1)
torch.rand(5, generator=gen1)
torch.rand(5, generator=gen1)
torch.rand(5, generator=gen0)
torch.rand(5, generator=gen0)


Instruction: How are contents of hidden_states tuple in BertModel in the transformers library arranged

Input:
model = BertModel.from_pretrained('bert-base-uncased', config=BertConfig.from_pretrained('bert-base-uncased',output_hidden_states=True))
outputs = model(input_ids)
hidden_states = outputs[2]

hidden_states is a tuple of 13 torch.FloatTensors. Each tensor is of size: (batch_size, sequence_length, hidden_size).
According to the documentation, the 13 tensors are the hidden states of the embedding and the 12 encoder layers.

My question:

Is hidden_states[0] the embedding layer while hidden_states[12] is the 12th encoder layer or

Is hidden_states[0] the embedding layer while hidden_states[12] is the 1st encoder layer or

Is hidden_states[0] the 12th encoder layer while hidden_states[12] is the embedding layer or

Is hidden_states[0] the 1st encoder layer while hidden_states[12] is the embedding layer

I havent found this found clearly stated anywhere else.

Output: Looking at the source-code for BertModel, it can be concluded that hidden_states[0] contains the outputs of the initial embedding layer, and the rest of the elements in tuples contain the hidden states in the increasing order of each layer. Simply put, hidden_states[1] contains the outputs of the first layer of BERT and hidden_states[12] contains the last i.e. 12th layer.


Instruction: How to solve ' CUDA out of memory. Tried to allocate xxx MiB' in pytorch?

Input:
I am trying to train a CNN in pytorch,but I meet some problems.
The RuntimeError:

  RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0;
  2.00 GiB total capacity; 584.97 MiB already allocated; 13.81 MiB free; 590.00 MiB reserved in total by PyTorch)

This is my code:

import os
import numpy as np
import cv2
import torch as t
import torch.nn as nn
import torchvision.transforms as transforms
from torch.utils.data import DataLoader,Dataset
import time
import matplotlib.pyplot as plt
%matplotlib inline
root_path='C:/Users/60960/Desktop/recet-task/course_LeeML20/course_LeeML20-datasets/hw3/food-11'
training_path=root_path+'/training'
testing_path=root_path+'/testing'
validation_path=root_path+'/validation'
def readfile(path,has_label):
    img_paths=sorted(os.listdir(path))
    x=np.zeros((len(img_paths),128,128,3),dtype=np.uint8)
    y=np.zeros((len(img_paths)),dtype=np.uint8)
    for i,file in enumerate(img_paths):
        img=cv2.imread(path+'/'+file)
        x[i,:,:]=cv2.resize(img,(128,128))
        if has_label:
            y[i]=int(file.split('_')[0])
    if has_label:
        return x,y
    else:
        return x
def show_img(img_from_cv2):
    b,g,r=cv2.split(img_from_cv2)
    img=cv2.merge([r,g,b])
    plt.imshow(img)
    plt.show()
x_train,y_train=readfile(training_path,True)
x_val,y_val=readfile(validation_path,True)
x_test=readfile(testing_path,False)
train_transform=transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(15),
    transforms.ToTensor()
])
test_transform=transforms.Compose([
    transforms.ToPILImage(),
    transforms.ToTensor()
])
class ImgDataset(Dataset):
    def __init__(self,x,y=None,transform=None):
        self.x=x
        self.y=y
        if y is not None:
            self.y=t.LongTensor(y)
        self.transform=transform
    def __len__(self):
        return len(self.x)
    def __getitem__(self,idx):
        X=self.x[idx]
        if self.transform is not None:
            X=self.transform(X)
        if self.y is not None:
            Y=self.y[idx]
            return X,Y
        return X
batch_size=128
train_set=ImgDataset(x_train,y_train,transform=train_transform)
val_set=ImgDataset(x_val,y_val,transform=test_transform)
train_loader=DataLoader(train_set,batch_size=batch_size,shuffle=True)
val_loader=DataLoader(val_set,batch_size=batch_size,shuffle=False)
class Classifier(nn.Module):
    def __init__(self):
        super(Classifier,self).__init__()
        self.cnn=nn.Sequential(
            nn.Conv2d(3,64,3,1,1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0),

            nn.Conv2d(64,128,3,1,1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0),

            nn.Conv2d(128,256,3,1,1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0),

            nn.Conv2d(256,512,3,1,1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0),

            nn.Conv2d(512,512,3,1,1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0)
        )
        self.fc=nn.Sequential(
            nn.Linear(512*4*4,1024),
            nn.ReLU(),
            nn.Linear(1024,512),
            nn.ReLU(),
            nn.Linear(512,11)
        )
    def forward(self,x):
        out=self.cnn(x)
        out=out.view(out.size()[0],-1)
        return self.fc(out)
model=Classifier().cuda()
loss_fn=nn.CrossEntropyLoss()
optim=t.optim.Adam(model.parameters(),lr=0.001)
epochs=30
for epoch in range(epochs):
    epoch_start_time=time.time()
    train_acc=0.0
    train_loss=0.0
    val_acc=0.0
    val_loss=0.0
    model.train()
    for i,data in enumerate(train_loader):
        optim.zero_grad()
        train_pred=model(data[0].cuda())
        batch_loss=loss_fn(train_pred,data[1].cuda())
        batch_loss.backward()
        optim.step()
        train_acc+=np.sum(np.argmax(train_pred.cpu().data.numpy(),axis=1)==data[1].numpy())
        train_loss+=batch_loss.item()
    model.eval()
    with t.no_grad():
        for i,data in enumerate(val_loader):
            val_pred=model(data[0].cuda())
            batch_loss=loss_fn(val_pred,data[1].cuda())
            val_acc+=np.sum(np.argmax(val_pred.cpu().data.numpy(),axis=1)==data[1].numpy())
            val_loss+=batch_loss.item()
        print('[%03d/%03d] %2.2f sec(s) Train Acc: %3.6f Loss: %3.6f | Val Acc: %3.6f loss: %3.6f' % (epoch + 1, epochs, time.time()-epoch_start_time,train_acc/train_set.__len__(), train_loss/train_set.__len__(), val_acc/val_set.__len__(), val_loss/val_set.__len__()))
x_train_val=np.concatenate((x_train,x_val),axis=0)
y_train_val=np.concatenate((y_train,y_val),axis=0)
train_val_set=ImgDataset(x_train_val,x_train_val,train_transform)
train_val_loader=DataLoader(train_val_set,batch_size=batch_size,shuffle=True)
model_final=Classifier().cuda()
loss_fn=nn.CrossEntropy()
optim=t.optim.Adam(model_final.parameters(),lr=0.001)
epochs=30
for epoch in range(epochs):
    epoch_start_time=time.time()
    train_acc=0.0
    train_loss=0.0
    model_final.train()
    for i,data in enumerate(train_val_loader):
        optim.zero_grad()
        train_pred=model_final(data[0].cuda())
        batch_loss=loss_fn(train_pred,data[1].cuda())
        batch_loss.backward()
        optim.step()
        train_acc+=np.sum(np.argmax(train_pred.cpu().data.numpy(),axis=1)==data[1].numpy())
        train_loss+=batch_loss.item()
    print('[%03d/%03d] %2.2f sec(s) Train Acc: %3.6f Loss: %3.6f' % (epoch + 1, epochs, time.time()-epoch_start_time,train_acc/train_val_set.__len__(), train_loss/train_val_set.__len__()))
test_set=ImgDataset(x_test,transform=test_transform)
test_loader=DataLoader(test_set,batch_size=batch_size,shuffle=False)
model_final.eval()
prediction=[]
with t.no_grad():
    for i,data in enumerate(test_loader):
        test_pred=model_final(data.cuda())
        test_label=np.argmax(test_pred.cpu().data.numpy(),axis=1)
        for y in test_label:
            prediction.append(y)
with open('predict.csv','w') as f:
    f.write('Id,Category\n')
    for i,y in enumerate(prediction):
        f.write('{},{}\n,'.format(i,y))

Pytorch version is 1.4.0, opencv2 version is 4.2.0.
The training dataset are pictures like these:training set

The error happens at this line:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-770be67177f4> in <module>
    119     for i,data in enumerate(train_loader):
    120         optim.zero_grad()
--> 121         train_pred=model(data[0].cuda())
    122         batch_loss=loss_fn(train_pred,data[1].cuda())
    123         batch_loss.backward()

I have already installed:
some information.
GPU utilization is low,close to zero:
GPU utilization.
Error message says:

  RuntimeError: CUDA out of memory.  Tried to allocate 512.00 MiB.

So I want to know how to allocate more memory.
What's more, I have tried to reduce the batch size to 1, but this doesn't work.
HELP!!!

Output:
Before reducing the batch size check the status of GPU memory :slight_smile:
nvidia-smi
Then check which process is eating up the memory choose PID and kill :boom: that process with
sudo kill -9 PID
or
sudo fuser -v /dev/nvidia*
sudo kill -9 PID


Instruction: Pytorch - TypeError: ToTensor() takes no arguments using torchvision.transform

Input:
I’m trying to load in a dataset for super-resolution and I have set up two functions which use Compose to crop and resize the images.

The function I have created for the input images works correctly and they are outputting as expected. The transform function for the target images is basically identical, just omitting the resize part of it.

def input_trans(c_size, sF):
    return Compose([
        CenterCrop(c_size),
        Resize(c_size // sF),
        ToTensor(),
    ])

def goal_trans(c_size):
    return Compose([
        CenterCrop(c_size),
        ToTensor(),
    ])

These functions are used in my dataset class when the images are loaded. I originally had goal = input.Copy() but I have changed it so both input and goal load the image separately. (was testing if the .copy() was the issue

def __getitem__(self, idx):
    input = Image.open(self.image_filenames[idx]).convert('RGB')
    goal = Image.open(self.image_filenames[idx]).convert('RGB')
    if self.input_transform:
        input = self.input_transform(input)
    if self.goal_transform:
        print(goal)
        print(goal.size)
        goal = self.goal_transform(goal)

    return input, goal

The error I receive is the following:

Traceback (most recent call last):
  File "main.py", line 128, in <module>
    main()  # execute this only when run directly, not when imported!
  File "main.py", line 55, in main
    train_model(epoch)
  File "main.py", line 40, in train_model
    for data_item, batch in enumerate(training_data_loader):
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "main.py", line 118, in __getitem__
    goal = self.goal_transform(goal)
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torchvision\transforms\transforms.py", line 70, in __call__
    img = t(img)
TypeError: ToTensor() takes no arguments

Confuses me because it doesn’t seem to have a problem with the first transformation (Ive checked and it does output before crashing).

I would appreciate any help you guys can give,

Thanks :)

Output:
Problem solved! nothing to do with torchvision.transforms.

I wasn't actually using the functions above, but inline declarations for compose which I had tried to use previously.

My bad


Instruction: Can auto-encoder encode new vector without re-training afresh?

Input:
Here is a simple autoencoder to encode 3 vectors of dimension 1x3 : [1,2,3],[1,2,3],[100,200,500] to 1x1 :

epochs = 1000
from pylab import plt
plt.style.use('seaborn')
import torch.utils.data as data_utils
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available()
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
import numpy as np
import pandas as pd
import datetime as dt

features = torch.tensor(np.array([ [1,2,3],[1,2,3],[100,200,500] ]))

print(features)

batch = 1
data_loader = torch.utils.data.DataLoader(features, batch_size=2, shuffle=False)

encoder = nn.Sequential(nn.Linear(3,batch), nn.Sigmoid())
decoder = nn.Sequential(nn.Linear(batch,3), nn.Sigmoid())
autoencoder = nn.Sequential(encoder, decoder)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.001)

encoded_images = []
for i in range(epochs):
    for j, images in enumerate(data_loader):
    #     images = images.view(images.size(0), -1)
        images = Variable(images).type(FloatTensor)
        optimizer.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()

#     encoded_images.append(encoder(images))

# print(decoder(torch.tensor(np.array([1,2,3])).type(FloatTensor)))

encoded_images = []
for j, images in enumerate(data_loader):
    images = images.view(images.size(0), -1)
    images = Variable(images).type(FloatTensor)

    encoded_images.append(encoder(images))

The variable encoded_images is an array of size 3 where each array entry represents the reduced dimensionality of a feature array :

[tensor([[0.9972],
         [0.9972]], grad_fn=<SigmoidBackward>),
 tensor([[1.]], grad_fn=<SigmoidBackward>)]

In order to determine similarity of a new feature, for example [1,1,1] is it required to re-train the network or can the existing trained network configuration/weights be "bootstrapped" such that the new vector can be encoded without require to retrain the network afresh ?

Output:
Sorry but your code is a mess... And if it's just to showcase the autoencoder idea (here you just have X, Y, Z coordinates while you name it image) it's chosen pretty poorly.

Out of the way: If it's an image you won't be able to encode it as a single pixel, this needs a little more sophistication.

Source code

  Here is a simple autoencoder to encode 3 vectors of dimension 1x3 :
  [1,2,3],[1,2,3],[100,200,500] to 1x1

Which is true only in this case as you have batch of 3 elements (while you named batch out_features of the network!). Their dimensions are not 1x3, it's just 3 as well. Here is a Minimal Reproducible Example with commentary:

import torch

# Rows are batches, there could be 3, there could be a thousand
data = torch.tensor([[1, 2, 3], [1, 2, 3], [100, 200, 500]]).float()

# 3 input features, columns of data
encoder = torch.nn.Sequential(torch.nn.Linear(3, 1), torch.nn.Sigmoid())
decoder = torch.nn.Sequential(torch.nn.Linear(1, 3), torch.nn.Sigmoid())

autoencoder = torch.nn.Sequential(encoder, decoder)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.001)

epochs = 10000

for i in range(epochs):
    optimizer.zero_grad()
    reconstructions = autoencoder(data)
    loss = torch.dist(data, reconstructions)
    loss.backward()
    optimizer.step()
    # Print loss every 100 epoochs
    if i % 100 == 0:
        print(loss)

Will it work?

This one is more interesting. In principle, if your neural network is trained, you don't have to retrain it to include example it didn't previously see (as the goal of neural network is to learn some patterns to solve the task).

In your case it won't.

Why it won't work?

First of all, you have sigmoid activation in decoder which restricts output to [0, 1] range. You are trying to predict data which is outside this range so it's impossible.

Without running, I can tell you what's the loss of this example will go towards to (with all weights being +inf). All predictions will be always [1, 1, 1] (or as close to it as possible) as this value penalizes the network the least, so you just have to calculate distance of each vector in data to [1, 1, 1]. Here loss is stuck around 546.2719. Weights and biases are around 10 (which is pretty huge for sigmoid) after 100000 epochs. Your values may vary but the trend is clear (though it will stop, as 10 is pretty close to 1 when you squash it with sigmoid).

Removing torch.nn.Sigmoid from decoder

What if we remove torch.nn.Sigmoid() from decoder? It will learn to almost perfectly reconstruct just your 3 examples, with loss being 0.002 after "only" 500000 epochs:

Here are the learned weights of decoder:

tensor([[ 99.0000],
        [198.0000],
        [496.9999]], requires_grad=True)

And here is the bias:

tensor([1.0000, 2.0000, 2.9999])

And here is the output of encoder for each example:

tensor([[2.2822e-13],
        [2.2822e-13],
        [1.0000e+00]])

Analysis of results

Your network learned just what you told it to learn, which is... magnitude (+ clever bias hackery).

[1, 2, 3] vector

Take [1, 2, 3] example (repeated twice). encoding of it is 2e-13 and goes towards zero, so we will assume it's zero.

Now, multiply 0 with all the weights, you still get zero. Add bias which is [1.0, 2.0, 2.99999] and you magically got your input reconstructed.

[100, 200, 500] vector

You can probably see where it's going.

Encoded value is 1.0, when multiplied by decoder weights we get [99.0, 198.0, 497.0]. Add bias to it and voila, we get our [100.0, 200.0, 500.0].

[1, 1, 1] vector

In your case it obviously will not work as magnitude of [1, 1, 1] is really small, hence it will be encoded as zero and reconstruced as [1, 2, 3].

Removing torch.nn.Sigmoid from encoder

A little off-topic, but when you remove sigmoid from encoder it won't be able to learn this pattern as "easily". The reason is the network has to be more conservative with the weights (as those won't be squashed). You would have to drop the learning rate (preferably constantly lowering it as the training progresses) as it becomes unstable at some point (when trying to hit "the perfect spot").

Learning similarity

It's hard (at least for the network) to define "similar" in this case. Is [1, 2, 3] similar to [3, 2, 1]? It has no concept of different dimensions and is required to squash those three numbers into a single value (later used for reconstruction).

As demonstrated, it would probably learn some implicit patterns in your data to be good at reconstructing "at least something", but won't find the general pattern you are looking for. Still it depends on your data and it's properties but I would argue for no in general and I think it's generalization capabilities would be poor.

And as you've seen in the analysis above, neural network is pretty good at finding those patterns even when you didn't see them (or maybe you did and that's what you were after?) or they don't exist at all.

If you need dimension similarity (and it's not just a thought experiment), you have a lot of "human-made" stuff like the p-norm, some encodings (those also measure similiarity but in a different way) so it's better to go for that IMO.


Instruction: About using RNN in pytorch

Input:
I am trying to use RNN to do a binary classification. But when my model is training, it gets stuck at loss.backward().
Here is my model:

class RNN2(nn.Module):
    def __init__(self, input_size, hidden_size, output_size=2, num_layers=1):
        super(RNN2, self).__init__()
        self.rnn = nn.RNN(input_size, hidden_size, num_layers)
        self.reg = nn.Linear(hidden_size, output_size)
        #self.softmax = nn.LogSoftmax(dim=1)

    def forward(self,x):
        x, hidden = self.rnn(x)
        return self.reg(x[:,2])

rnn = RNN2(13,10)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)
for e in range(10):
    out = rnn(train_X)
    optimizer.zero_grad()
    print(out[0])
    print(out.shape)
    print(train_Y.shape)
    loss = criterion(out, train_Y)
    print(loss)
    loss.backward()
    print("1")
    optimizer.step()
    print("2")

The shape of train_X is 420000*3*13 and the shape of train_Y is 420000
So it can print loss. Can anyone tell me why it gets stuck at loss.backward(). It can't print 1.

Output:
You have to know that in RRNs, computing the backward function for a sequence of length 420000 is extremely slow. If you run your code on a machine with a GPU (or google colab) and add the following lines before the for loop, your code finishes executing in less than two minutes.

rnn = rnn.cuda()
train_X = train_X.cuda()
train_Y = train_Y.cuda()

Note that by default, the second input dimension passed to RNN will be treated as the batch size. Therefore, if the 420000 is the number of batches, pass batch_first=True to the RNN constructor.

self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True)

This would significantly speed up the process (less than one second in google colab). However, if that is not the case, you should try chunking the sequences into smaller parts and increasing the batch size from 3 to a larger value.


Instruction: TensorBoard: Tutorial Pytorch: module 'tensorflow._api.v2.io.gfile' has no attribute 'get_filesystem'

Input:
I am having issues running this tutorial about Pytorch and TensorBoard with Embeddings https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html

I am having this message,

AttributeError                            Traceback (most recent call last)
<ipython-input-10-e0404d94b4cd> in <module>()
     22 writer.add_embedding(features,
     23                     metadata=class_labels,
---> 24                     label_img=images.unsqueeze(1))
     25 writer.close()

/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/writer.py in add_embedding(self, mat, metadata, label_img, global_step, tag, metadata_header)
    779         save_path = os.path.join(self._get_file_writer().get_logdir(), subdir)
    780
--> 781         fs = tf.io.gfile.get_filesystem(save_path)
    782         if fs.exists(save_path):
    783             if fs.isdir(save_path):

and I can visualize the projector

Output:
Try this

import tensorflow as tf
import tensorboard as tb
tf.io.gfile = tb.compat.tensorflow_stub.io.gfile


Instruction: Is it possible to use LSTM predictions as inputs for next time steps?

Input:
I am working with LSTM (in PyTorch) for multivariate time series prediction. Let’s imagine the situation: I have 2 time series, A and B, and I want to predict t-value of B using previous values of A and B (before t). Such prediction works fine, my model gets good results.

But what if (during testing, after training) I want to use predicted values of B as inputs for next time step instead of real values? For example: I predict first value of B, make a step, put predicted value instead of a real, and make prediction again. Then I use two predicted values instead of real two, and so on. In some steps only predicted values will be in time series B.

Are there any possibilities to do that?

Output:
This is exactly what people do for machine translation and text generation in general. In this case, the LSTM predicts a distribution over a vocabulary, you select one word and use it as an input to the network in the next step. See PyTotrch tutorial on machine translation for more details.

The important point is that the LSTM executed in two regimes:

For training: as a standard sequence labeling. It is provided input and it should predict one step in the future.
For inference: It gradually generates new samples and uses them as the next input. In PyTorch, this needs to be implemented with an explicit for loop.


Instruction: How to visualize a torch_geometric graph in Python?

Input:
Let's consider as an example that I have the following adjacence matrix in coordinate format:

> edge_index.numpy() = array([[    0,     1,     0,   3,   2],
                              [    1,     0,     3,   2,   1]], dtype=int64)

which means that the node 0 is linked toward the node 1, and vice-versa, the node 0 is linked to 3 etc...

Do you know a way to draw this graph as in networkx with nx.draw() ?

Thank you.

Output:
import networkx as nx

edge_index = torch.tensor([[0, 1, 1, 2],
                           [1, 0, 2, 1]], dtype=torch.long)
x = torch.tensor([[-1], [0], [1]], dtype=torch.float)

data = torch_geometric.data.Data(x=x, edge_index=edge_index)
g = torch_geometric.utils.to_networkx(data, to_undirected=True)
nx.draw(g)


Instruction: BERT token importance measuring issue. Grad is none

Input:
I am trying to measure token importance for BERT via comparing token embedding grad value. So, to get the grad, I've copied the 2.8.0 forward of BertModel and changed it a bit:

huggingface transformers 2.8.0 BERT https://github.com/huggingface/transformers/blob/11c3257a18c4b5e1a3c1746eefd96f180358397b/src/transformers/modeling_bert.py

Code:

        embedding_output = self.embeddings(
            input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
        )
        embedding_output = embedding_output.requires_grad_(True) # my code
        encoder_outputs = self.encoder(
            embedding_output,
            attention_mask=extended_attention_mask,
            head_mask=head_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_extended_attention_mask,
        )
        sequence_output = encoder_outputs[0]
        sequence_output.mean().backward() # my code
        assert(embedding_output.grad is not None) # my code

Colab link: https://colab.research.google.com/drive/1MggBUaDWAAZNuXbTDM11E8jvdMGEkuRD
But it gives assertion error. I do not understand why and it seems to be a bug for me.
Please, help!

Output:
I needed to add this line:

embedding_output = torch.tensor(embedding_output, requires_grad=True)

It seems, that I used .requires_grad_ method incorrectly.


Instruction: Why not super().__init__(Model,self) in Pytorch

Input:
For torch.nn.Module()

According to the official documentation:
Base class for all neural network modules.
Your models should also subclass this class.
Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes.

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

It used super(Model, self).__init__()
Why not super().__init__(Model, self)

Output:
This construct:
super().__init__(self)

is valid only in Python 3.x whereas the following construct,
super(Model, self).__init__()

works both in Python 2.x and Python 3.x. So, the PyTorch developers didn't want to break all the code that's written in Python 2.x by enforcing the Python 3.x syntax of super() since both constructs essentially do the same thing in this case, which is initializing the following variables:
    self.training = True
    self._parameters = OrderedDict()
    self._buffers = OrderedDict()
    self._backward_hooks = OrderedDict()
    self._forward_hooks = OrderedDict()
    self._forward_pre_hooks = OrderedDict()
    self._state_dict_hooks = OrderedDict()
    self._load_state_dict_pre_hooks = OrderedDict()
    self._modules = OrderedDict()

For details, see the relevant discussion in the PyTorch forum on the topic, is-there-a-reason-why-people-use-super-class-self-init-instead-of-super-init?


Instruction: Finding memory leak in python by tracemalloc module

Input:
I have a python script which uses an opensource pytorch model and this code has a memory leak. I am running this with memory_profiler mprof run --include-children python my_sctipt.py and get the following image:

I am trying to search for the reason of the leak by the system python module tracemalloc:

tracemalloc.start(25)
while True:
    ...
    snap = tracemalloc.take_snapshot()
    domain_filter = tracemalloc.DomainFilter(True, 0)
    snap = snap.filter_traces([domain_filter])
    stats = snap.statistics('lineno', True)
    for stat in stats[:10]:
        print(stat)

If looking only at tracemalloc output, I will not be able to identify the problem. I assume that the problem is in the C extension but, I would like to make sure it is true.
I tried to change the domain by DomainFilter, but I have output only in 0 domain.

Also, I don't understand the meaning of the parameter which tracemalloc.start(frameno) has got, frameno is a number of the most recent frames, but nothing happens when I change it.

What can I do next to find the problematic place in the code which causes the memory leak?

Looking forward to your answer.

Output:
Given that your guess is that the problem is in the C extension, but that you want to make sure this is true, I would suggest that you do so using a tool that is less python-specific like  https://github.com/vmware/chap or at least if you are able to run your program on Linux.

What you will need to do is run your script (uninstrumented) and at some point gather a live core (for example, using "gcore pid-of-your-running-program").

Once you have that core, open that core in chap ("chap your-core-file-path") and try the following command from the chap prompt:

summarize writable

The output will be something like this, but your numbers will likely vary considerably:

chap> summarize writable
5 ranges take 0x2021000 bytes for use: stack
6 ranges take 0x180000 bytes for use: python arena
1 ranges take 0xe1000 bytes for use: libc malloc main arena pages
4 ranges take 0x84000 bytes for use: libc malloc heap
8 ranges take 0x80000 bytes for use: used by module
1 ranges take 0x31000 bytes for use: libc malloc mmapped allocation
4 ranges take 0x30000 bytes for use: unknown
29 writable ranges use 0x23e7000 (37,646,336) bytes.

The lines in the summary are given in decreasing order of byte usage, so you can follow that order.  So looking at the top one first we see that the use is "stack":

5 ranges take 0x2021000 bytes for use: stack

This particular core was for a very simple python program that starts 4 extra threads and has all 5 threads sleep.  The reason large stack allocations can happen rather easily with a multi-threaded python program is that python uses pthreads to create additional threads and pthreads uses the ulimit value for stack size as a default.  If your program has a similarly large value, you can change the stack size in one of several ways, including running "ulimit -s" in the parent process to change the default stack size.  To see what values actually make sense you can use the following command from the chap prompt:

chap> describe stacks
Thread 1 uses stack block [0x7fffe22bc000, 7fffe22dd000)
 current sp: 0x7fffe22daa00
Peak stack usage was 0x7798 bytes out of 0x21000 total.

Thread 2 uses stack block [0x7f51ec07c000, 7f51ec87c000)
 current sp: 0x7f51ec87a750
Peak stack usage was 0x2178 bytes out of 0x800000 total.

Thread 3 uses stack block [0x7f51e7800000, 7f51e8000000)
 current sp: 0x7f51e7ffe750
Peak stack usage was 0x2178 bytes out of 0x800000 total.

Thread 4 uses stack block [0x7f51e6fff000, 7f51e77ff000)
 current sp: 0x7f51e77fd750
Peak stack usage was 0x2178 bytes out of 0x800000 total.

Thread 5 uses stack block [0x7f51e67fe000, 7f51e6ffe000)
 current sp: 0x7f51e6ffc750
Peak stack usage was 0x2178 bytes out of 0x800000 total.

5 stacks use 0x2021000 (33,689,600) bytes.

So what you see above is that 4 of the stacks are 8MiB in size but could easily be well under 64KiB.

Your program may not have any issues with stack size, but if so, you can fix them as described above.

Continuing with checking for causes of growth, look at the next line from the summary:

6 ranges take 0x180000 bytes for use: python arena

So python arenas use the next most memory.  These are used strictly for python-specific allocations.  So if this value is large in your case it disproves your theory about C allocations being the culprit, but there is more you can do later to figure out how those python allocations are being used.

Looking at the remaining lines of the summary, we see a few with "libc" as part of the "use" description:

1 ranges take 0xe1000 bytes for use: libc malloc main arena pages
4 ranges take 0x84000 bytes for use: libc malloc heap
1 ranges take 0x31000 bytes for use: libc malloc mmapped allocation

Note that libc is responsible for all that memory, but you can't know that the memory is used for non-python code because for allocations beyond a certain size threshold (well under 4K) python grabs memory via malloc rather than grabbing memory from one of the python arenas.

So lets assume that you have resolved any issues you might have had with stack usage and you have mainly "python arenas" or "libc malloc" related usages.  The next thing you want to understand is whether that memory is mostly "used" (meaning allocated but never freed) or "free" (meaning "freed but not given back to the operating system).  You can do that as shown:

chap> count used
15731 allocations use 0x239388 (2,331,528) bytes.
chap> count free
1563 allocations use 0xb84c8 (754,888) bytes.

So in the above case, used allocations dominate and what one should do is to try to understand those used allocations.  The case where free allocations dominate is much more complex and is discussed a bit in the user guide but would take too much time to cover here.

So lets assume for now that used allocations are the main cause of growth in your case.  We can find out why we have so many used allocations.

The first thing we might want to know is whether any allocations were actually "leaked" in the sense that they are no longer reachable.  This excludes the case where the growth is due to container-based growth.

One does this as follows:

chap> summarize leaked
0 allocations use 0x0 (0) bytes.

So for this particular core, as is pretty common for python cores, nothing was leaked.  Your number may be non-zero.  If it is non-zero but still much lower than the totals associated with memory used for "python" or "libc" reported above, you might just make a note of the leaks but continue to look for the real cause of growth.  The user guide has some information about investigating leaks but it is a bit sparse.  If the leak count is actually big enough to explain your growth issue, you should investigate that next but if not, read on.

Now that you are assuming container-based growth the following commands are useful:

chap> redirect on
chap> summarize used
Wrote results to scratch/core.python_5_threads.summarize_used
chap> summarize used /sortby bytes
Wrote results to scratch/core.python_5_threads.summarize_used::sortby:bytes

The above created two text files, one which has a summary ordered in terms of object counts and another which has a summary in terms of the total bytes used directly by those objects.

At present chap has only very limited support for python (it finds those python objects, in addition to any allocated by libc malloc but for python objects the summary only breaks out limited categories for python objects in terms of patterns (for example, %SimplePythonObject matches things like "int", "str", ... that don't hold other python objects and %ContainerPythonObject matches things like tuple, list, dict, ... that do hold references to other python objects).  With that said, it should be pretty easy to tell from the summary whether the growth in used allocations is primarily due to objects allocated by python or objects allocated by native code.

So in this case, given that you specifically are trying to find out whether the growth is due to native code or not, look in the summary for counts like the following, all of which are python-related:

Pattern %SimplePythonObject has 7798 instances taking 0x9e9e8(649,704) bytes.

Pattern %ContainerPythonObject has 7244 instances taking 0xc51a8(807,336) bytes.

Pattern %PyDictKeysObject has 213 instances taking 0xb6730(747,312) bytes.

So in the core I have been using for an example, definitely python allocations dominate.

You will also see a line for the following, which is for allocations that chap does not yet recognize.  You can't make assumptions about whether these are python-related or not.

Unrecognized allocations have 474 instances taking 0x1e9b8(125,368) bytes.

This will hopefully answer your basic question of what you can do next.  At least at that point you will understand whether the growth is likely due to C code or python code and depending on what you find, the chap user guide should help you go further from there.


Instruction: PyTorch: why the difference between dir(nn.Module()) and dir(nn.Module)

Input:
Tried

e = dir(nn.Module())
f = dir(nn.Module)
print([item for item in e if item not in f])

It gives

['_backward_hooks', '_buffers', '_forward_hooks', '_forward_pre_hooks', '_load_state_dict_pre_hooks', '_modules', '_parameters', '_state_dict_hooks', 'training']

why these are only available for the object not class ?
What's special about these attributes?

Output:
It's the other way around, those attributes are only available on the object (e
 in your case), but not on the class. The reason is simple, those are the attributes that are created in the constructor, hence they don't exist on the class and are only created when the object is created.

From the nn.Module implementation:

def __init__(self):
    """
    Initializes internal Module state, shared by both nn.Module and ScriptModule.
    """
    torch._C._log_api_usage_once("python.nn_module")

    self.training = True
    self._parameters = OrderedDict()
    self._buffers = OrderedDict()
    self._backward_hooks = OrderedDict()
    self._forward_hooks = OrderedDict()
    self._forward_pre_hooks = OrderedDict()
    self._state_dict_hooks = OrderedDict()
    self._load_state_dict_pre_hooks = OrderedDict()
    self._modules = OrderedDict()

There is nothing special about these attributes, they are simply used to track the state of each individual module.


Instruction: The Batch Size of batch normalisation and gradient descent

Input:
We need to choose a batch size for GD as well as the normalization, they both called batch size, but in actual implementation, do they need to be the same?
Or otherwise how the framework handle them?

In Pytorch for example, one batch size is defined in dataloader, e.g.

torch.utils.data.DataLoader(image_datasets[x],
batch_size=16,
shuffle=True,
num_workers=4)

and if use BN

self.bn1 = nn.BatchNorm2d(16)

Does it has to be the same (16) or it can be different ? If different, is there any preferred relationship between the two 'batch size'
Thanks

Output:
No, batch_size is only defined in the data loader, not in the model.

The BatchNorm2d has a num_features parameter, and it depends on the number of channels and not the batch size, as you can see in the docs.

They are completely unrelated.

  BatchNorm2d

  torch.nn.BatchNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

  Parameters

  num_features – C from an expected input of size (N,C,H,W)(N, C, H, W)(N,C,H,W)
  eps – a value added to the denominator for numerical stability. Default: 1e-5
  momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1
  affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True
  track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True


Instruction: pytorch geometric "Detected that PyTorch and torch_sparse were compiled with different CUDA versions" on google colab

Input:
I'm new to pytorch geometric, tried to install it to my computer but failed, so I'm trying to run the code on Google Colab instead. According to this previous question (which didn't help me and I'mnot sure its the same issue):

PyTorch Geometric CUDA installation issues on Google Colab

I did:

!pip install --upgrade torch-scatter

!pip install --upgrade torch-sparse

!pip install --upgrade torch-cluster

!pip install --upgrade torch-spline-conv

!pip install torch-geometric

!pip install torch-cluster==latest+cu101 -f https://s3.eu-central-1.amazonaws.com/pytorch-geometric.com/whl/torch-1.4.0.html

!pip install torch-scatter==latest+cu101 torch-sparse==latest+cu101 torch-spline-conv==latest+cu101 -f https://s3.eu-central-1.amazonaws.com/pytorch-geometric.com/whl/torch-1.4.0.html

they print:

Successfully installed torch-cluster-1.5.4

Successfully installed torch-scatter-2.0.4 torch-sparse-0.6.1 torch-spline-conv-1.2.0

However, when I try to run

import torch_geometric.datasets as datasets

I get:

RuntimeError: Detected that PyTorch and torch_sparse were compiled with different CUDA versions. PyTorch has CUDA version 10.1 and torch_sparse has CUDA version 0.0. Please reinstall the torch_sparse that matches your PyTorch install.

Any help would be greatly appretiated.

Output:
I came up with the following snippet that should work on Colab to install PyTorch Geometric and its dependencies:
https://gist.github.com/ameya98/b193856171d11d37ada46458f60e73e7
# Add this in a Google Colab cell to install the correct version of Pytorch Geometric.
import torch

def format_pytorch_version(version):
  return version.split('+')[0]

TORCH_version = torch.__version__
TORCH = format_pytorch_version(TORCH_version)

def format_cuda_version(version):
  return 'cu' + version.replace('.', '')

CUDA_version = torch.version.cuda
CUDA = format_cuda_version(CUDA_version)

!pip install torch-scatter     -f https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html
!pip install torch-sparse      -f https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html
!pip install torch-cluster     -f https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html
!pip install torch-spline-conv -f https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html
!pip install torch-geometric


Instruction: From Coco annotation json to semantic segmentation image like VOC's .png in pytorch

Input:
I am trying to use COCO 2014 data for semantic segmentation training in PyTorch.
I have a PSPNet model with a Cross Entropy loss function that worked perfectly on PASCAL VOC dataset from 2012.
Now I am trying to use a portion of COCO pictures to do the same process. But Coco has json data instead of .png images for annotation and I somehow have to covert one to the other. I have noticed that there is annToMask in cocotools, but I cannot quiet figure out how to use that function in my case.
This is kind of what my dataloader's pull item looks like

def pull_item(self, index):

        I DONT KNOW WHAT TO DO HERE

        raw_img = self.transform(raw_img)
        anns_img = self.transform(anns_img)

        return raw_img, anns_img

Below is what my training function that uses data from dataloaders looks like.

 for images, labels in dataloaders_dict[phase]:

                images = images.to(device)

                labels = torch.squeeze(labels)
                labels = labels.to(device)

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = net(images)

                    loss = criterion(outputs, labels.long())

Output:
I have worked on creating a Data Generator for the COCO dataset with PyCOCO and I think my experience can help you out. My post on medium documents the entire process from start to finish, including the creation of masks.

However, point to note, I was working with Tensorflow Keras and not pytorch. But the logic flow should largely be the same, so I am sure you can take back something useful from it.


Instruction: How to understand hidden_states of the returns in BertModel?(huggingface-transformers)

Input:
  Returns last_hidden_state (torch.FloatTensor of shape (batch_size,
  sequence_length, hidden_size)): Sequence of hidden-states at the
  output of the last layer of the model.

  pooler_output (torch.FloatTensor: of shape (batch_size, hidden_size)):
  Last layer hidden-state of the first token of the sequence
  (classification token) further processed by a Linear layer and a Tanh
  activation function. The Linear layer weights are trained from the
  next sentence prediction (classification) objective during
  pre-training.

  This output is usually not a good summary of the semantic content of
  the input, you’re often better with averaging or pooling the sequence
  of hidden-states for the whole input sequence.

  hidden_states (tuple(torch.FloatTensor), optional, returned when
  config.output_hidden_states=True): Tuple of torch.FloatTensor (one for
  the output of the embeddings + one for the output of each layer) of
  shape (batch_size, sequence_length, hidden_size).

  Hidden-states of the model at the output of each layer plus the
  initial embedding outputs.

  attentions (tuple(torch.FloatTensor), optional, returned when
  config.output_attentions=True): Tuple of torch.FloatTensor (one for
  each layer) of shape (batch_size, num_heads, sequence_length,
  sequence_length).

  Attentions weights after the attention softmax, used to compute the
  weighted average in the self-attention heads.

This is from https://huggingface.co/transformers/model_doc/bert.html#bertmodel. Although the description in the document is clear, I still don't understand the hidden_states of returns. There is a tuple, one for the output of the embeddings, and the other for the output of each layer.
Please tell me how to distinguish them, or what is the meaning of them? Thanks very much!![wink~

Output:
hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True): Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

For a given token, its input representation is constructed by summing the corresponding token embedding, segment embedding, and position embedding. This input representation is called the initial embedding output which can be found at index 0 of the tuple hidden_states.
This figure explains how the embeddings are calculated.

The remaining 12 elements in the tuple contain the output of the corresponding hidden layer. E.g: the last hidden layer can be found at index 12, which is the 13th item in the tuple. The dimension of both the initial embedding output and the hidden states are [batch_size, sequence_length, hidden_size]. It would be useful to compare the indexing of hidden_states bottom-up with this image from the BERT paper.


Instruction: IndexError: Target -1 is out of bounds error in tabular learner fatai2

Input:
Getting the below error when trying to fit a tabular_learner from fastai2 library.

used data loaders

learn = tabular_learner(dls, layers=[1000,500], metrics=accuracy)

learn.fit(30,1e-2)

IndexError                                Traceback (most recent call last)
<ipython-input-35-f0c57ab3748f> in <module>
----> 1 learn.fit(30,1e-2)

/mnt/c/fastai2/fastai2/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    191                         self.epoch=epoch;          self('begin_epoch')
    192                         self._do_epoch_train()
--> 193                         self._do_epoch_validate()
    194                     except CancelEpochException:   self('after_cancel_epoch')
    195                     finally:                       self('after_epoch')

/mnt/c/fastai2/fastai2/learner.py in _do_epoch_validate(self, ds_idx, dl)
    173             dl,old,has = change_attrs(dl, names, [False,False])
    174             self.dl = dl;                                    self('begin_validate')
--> 175             with torch.no_grad(): self.all_batches()
    176         except CancelValidException:                         self('after_cancel_validate')
    177         finally:

/mnt/c/fastai2/fastai2/learner.py in all_batches(self)
    141     def all_batches(self):
    142         self.n_iter = len(self.dl)
--> 143         for o in enumerate(self.dl): self.one_batch(*o)
    144
    145     def one_batch(self, i, b):

/mnt/c/fastai2/fastai2/learner.py in one_batch(self, i, b)
    149             self.pred = self.model(*self.xb);                self('after_pred')
    150             if len(self.yb) == 0: return
--> 151             self.loss = self.loss_func(self.pred, *self.yb); self('after_loss')
    152             if not self.training: return
    153             self.loss.backward();                            self('after_backward')

/mnt/c/fastai2/fastai2/layers.py in __call__(self, inp, targ, **kwargs)
    291         if targ.dtype in [torch.int8, torch.int16, torch.int32]: targ = targ.long()
    292         if self.flatten: inp = inp.view(-1,inp.shape[-1]) if self.is_2d else inp.view(-1)
--> 293         return self.func.__call__(inp, targ.view(-1) if self.flatten else targ, **kwargs)
    294
    295 # Cell

~/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

~/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

~/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target -1 is out of bounds.

Any clue would be much appreciated! Thanks

Output:
Finally got figured this out, this happened because my validation set accidentally had more dependent variable classes than in my training set (or may be it was the other way around)......To fix this I had to ensure that my class size of training set and validation set are the same

i:e make sure you do this check

len(train_df["my_category"].unique()) == len(valid_df["my_category"].unique())


Instruction: Need help understanding this Python list syntax

Input:
I'm having trouble understanding what this syntax means in Python:

out = out[lengths - 1, range(len(lengths))]

Why is there a range inside a list? How does that work? For context, I'm training a machine learning model in PyTorch. lengths is a list of the lengths of the input.

Output:
I assume lengths is an array of integers. (probably a Numpy array)

The first index lengths - 1 will give a list of indices that is subtracted by -1.
The second index range(len(lengths)) will give a list of numbers from 0 to the size of lengths.

I don't know what the specific logic is in your code, but in general, you can give a list of indices to pick the data at specific locations.

out = np.array([[1,2,3],[4,5,6],[7,8,9]])
-> array([[1, 2, 3],
          [4, 5, 6],
          [7, 8, 9]])

first_idx = [0, 1, 2]
second_idx = [2, 1, 0]
# (0,2), (1,1), (2,0)

out[first_idx, second_idx]
-> array([3, 5, 7])


Instruction: PyTorch - sparse tensors do not have strides

Input:
I am building my first sentiment analysis model for a small dataset of 1000 reviews using TF-IDF approach along with LSTM using the below code. I am preparing the train data by preprocessing it and feeding to the Vectorizer as below

def tfidf_features(X_train, X_val, X_test):
tfidf_vectorizer = TfidfVectorizer(analyzer='word', token_pattern = '(\S+)', min_df = 5, max_df =
0.9, ngram_range=(1,2))
X_train=tfidf_vectorizer.fit_transform(X_train)
X_val=tfidf_vectorizer.transform(X_val)
X_test=tfidf_vectorizer.transform(X_test)

return X_train, X_val, X_test, tfidf_vectorizer.vocabulary_

I am converting my csr_matrix to a pytorch tensor using the below code

def spy_sparse2torch_sparse(data):
samples=data.shape[0]
features=data.shape[1]
values=data.data
coo_data=data.tocoo()
indices=torch.LongTensor([coo_data.row,coo_data.col])
t=torch.sparse.FloatTensor(indices,torch.from_numpy(values).float(),[samples,features])
return t

And I am getting the training sentences tensor as this

   tensor(indices=tensor([[  0,   0,   1,  ..., 599, 599, 599],
                   [ 97, 131,  49,  ..., 109,  65,  49]]),
   values=tensor([0.6759, 0.7370, 0.6076,  ..., 0.3288, 0.3927, 0.3288]),
   size=(600, 145), nnz=1607, layout=torch.sparse_coo)

I am creating a TensorDataSet using the below code wherein I am also converting my label data from bumpy to a torch tensor

train_data = TensorDataset(train_x, torch.from_numpy(train_y))

I have defined my LSTM network and calling it with the following parameters

n_vocab = len(vocabulary)
n_embed = 100
n_hidden = 256
n_output = 1   # 1 ("positive") or 0 ("negative")
n_layers = 2

net = Sentiment_Lstm(n_vocab, n_embed, n_hidden, n_output, n_layers)

I have also defined the loss and optimizer. Now I am training my model using the below code

print_every = 100
step = 0
n_epochs = 4  # validation loss increases from ~ epoch 3 or 4
clip = 5  # for gradient clip to prevent exploding gradient problem in LSTM/RNN

for epoch in range(n_epochs):
h = net.init_hidden(batch_size)

for inputs, labels in train_loader:
    step += 1

    # making requires_grad = False for the latest set of h
    h = tuple([each.data for each in h])

    net.zero_grad()
    output, h = net(inputs)
    loss = criterion(output.squeeze(), labels.float())
    loss.backward()
    nn.utils.clip_grad_norm(net.parameters(), clip)
    optimizer.step()

    if (step % print_every) == 0:
        net.eval()
        valid_losses = []
        v_h = net.init_hidden(batch_size)

        for v_inputs, v_labels in valid_loader:
            v_inputs, v_labels = inputs.to(device), labels.to(device)

            v_h = tuple([each.data for each in v_h])

            v_output, v_h = net(v_inputs)
            v_loss = criterion(v_output.squeeze(), v_labels.float())
            valid_losses.append(v_loss.item())

        print("Epoch: {}/{}".format((epoch+1), n_epochs),
              "Step: {}".format(step),
              "Training Loss: {:.4f}".format(loss.item()),
              "Validation Loss: {:.4f}".format(np.mean(valid_losses)))
        net.train()

However, I am getting a major error on the line output, h = net(inputs) as RuntimeError: sparse tensors do not have strides

The workarounds given on other websites are not understandable. I am expecting an exact code change I need to make in order to fix this issue.

Output:
Pytorch does not support sparse (S) to sparse matrix multiplication.
Let us consider :
torch.sparse.mm(c1,c2), where c1 and c2 are sparse_coo_tensor matrices.
case1: If we try c1 and c2 to be S  --> It gives the erros RuntimeError: sparse tensors do not have strides.
case2: If c1 is dense (D) and c2 is S --> It gives the same error.
case3: Only when c1 is S and c2 is D --> It works fine.
Reference: https://blog.csdn.net/w55100/article/details/109086131
I guess the matrix multiplication happening in your Sentiment_Lstm might be falling under the first two cases. And thereby throwing this error.
By using dense input format it should work.