Instruction: Using Softmax Activation function after calculating loss from BCEWithLogitLoss (Binary Cross Entropy + Sigmoid activation)

Input:
I am going through a Binary Classification tutorial using PyTorch and here, the last layer of the network is torch.Linear()  with just one neuron. (Makes Sense) which will give us a single neuron.  as pred=network(input_batch)

After that the choice of Loss function is loss_fn=BCEWithLogitsLoss()  (which is numerically stable than using the softmax first and then calculating loss) which will apply Softmax function to the output of last layer to give us a probability.  so after that, it'll calculate the binary cross entropy to minimize the loss.

loss=loss_fn(pred,true)

My concern is that after all this, the author used torch.round(torch.sigmoid(pred))

Why would that be? I mean I know it'll get the prediction probabilities in the range [0,1] and then round of the values with default threshold of 0.5.

Isn't it better to use the sigmoid once after the last layer within the network rather using a softmax and a sigmoid at 2 different places given it's a binary classification??

Wouldn't it be better to just

out = self.linear(batch_tensor)
return self.sigmoid(out)

and then calculate the BCE loss and use the argmax() for checking accuracy??

I am just curious that can it be a valid strategy?

Output:
You seem to be thinking of the binary classification as a multi-class classification with two classes, but that is not quite correct when using the binary cross-entropy approach. Let's start by clarifying the goal of the binary classification before looking at any implementation details.

Technically, there are two classes, 0 and 1, but instead of considering them as two separate classes, you can see them as opposites of each other. For example, you want to classify whether a StackOverflow answer was helpful or not. The two classes would be "helpful" and "not helpful". Naturally, you would simply ask "Was the answer helpful?", the negative aspect is left off, and if that wasn't the case, you could deduce that it was "not helpful". (Remember, it's a binary case, there is no middle ground).

Therefore, your model only needs to predict a single class, but to avoid confusion with the actual two classes, that can be expressed as: The model predicts the probability that the positive case occurs. In context of the previous example: What is the probability that the StackOverflow answer was helpful?

Sigmoid gives you values in the range [0, 1], which are the probabilities. Now you need to decide when the model is confident enough for it to be positive by defining a threshold. To make it balanced, the threshold is 0.5, therefore as long as the probability is greater than 0.5 it is positive (class 1: "helpful") otherwise it's negative (class 0: "not helpful"), which is achieved by rounding (i.e. torch.round(torch.sigmoid(pred))).

  After that the choice of Loss function is loss_fn=BCEWithLogitsLoss() (which is numerically stable than using the softmax first and then calculating loss) which will apply Softmax function to the output of last layer to give us a probability.

  Isn't it better to use the sigmoid once after the last layer within the network rather using a softmax and a sigmoid at 2 different places given it's a binary classification??

BCEWithLogitsLoss applies Sigmoid not Softmax, there is no Softmax involved at all. From the nn.BCEWithLogitsLoss documentation:

  This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.

By not applying Sigmoid in the model you get a more numerically stable version of the binary cross-entropy, but that means you have to apply the Sigmoid manually if you want to make an actual prediction outside of training.

  [...] and use the argmax() for checking accuracy??

Again, you're thinking of the multi-class scenario. You only have a single output class, i.e. output has size [batch_size, 1]. Taking argmax of that, will always give you 0, because that is the only available class.


Instruction: how to get the index of the subarray in pytorch?

Input:
a and b are torch tensor No repeating elements
a shape is [n,2] like:

[[1,2]
[2,3]
[4,6]
...]

b is[m,2] like:

[[1,2]
[4,6]
....
]

how to get the index of b in a, example:

a = [[1,2]
[2,4]
[6,7]
]
b = [[1,2]
[6,7]]

the index should be (0,3), we can use gpu，

Output:
Here @jpp 's, numpy solution is almost your answer after this

You just need to get indices using nonzero and flatten tensor using flatten to get expected shape.

a = torch.tensor([[1, 2], [2, 4], [6, 7]])
b = torch.tensor([[1, 2], [6, 7]])

(a[:, None] == b).all(-1).any(-1).nonzero().flatten()
tensor([0, 2])


Instruction: Print the values of torch.data.dataset object

Input:
I have made my pandas dataframe X_train into z tensor but the output is

user_id=torchtext.data.RawField()
fields=[('user_id',user_id)]
from torchtext.data import Dataset,Example
z=torchtext.data.Dataset(X_train.user_id,fields)
print(len(z))
print(z)

Ouput is:

426018
<torchtext.data.dataset.Dataset object at 0x7feffb6a8f98>

How should I print the actual data in my variable object z?

Output:
Could print(list(torch.utils.data.DataLoader())) be what you're looking for?

DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,
           batch_sampler=None, num_workers=0, collate_fn=None,
           pin_memory=False, drop_last=False, timeout=0,
           worker_init_fn=None)


Instruction: How conv1d pytorch operates on a sequence of characters or frames?

Input:
I understand convolution filters when applied to an image (e.g. an 224x224 image with 3 in-channels transformed by 56 total filters of 5x5 conv to a 224x224 image with 56 out-channels). The key is that there are 56 different filters each with 5x5x3 weights that end up producing output image 224x224, 56 (term after comma is output channels).

But I can't seem to understand how conv1d filter works in seq2seq models on a sequence of characters. One of the models i was looking at https://arxiv.org/pdf/1712.05884.pdf has a "post-net layer is comprised of 512 filters with shape 5×1" that operates on a spectrogram frame 80-d (means 80 different float values in the frame), and the result of filter is a 512-d frame.

I don't understand what in_channels, out_channels mean in pytorch conv1d definition as in images I can easily understand what in-channels/out-channels mean, but for sequence of 80-float values frames I'm at loss. What do they mean in the context of seq2seq model like this above?
How do 512, 5x1 filters on 80 float values produce 512 float values?**
Wouldn't a 5x1 filter when operating on 80 float values just produce 80 float values (by just taking 5 consecutive values at a time of those 80)? How many weights total these 512 filters have?**

The layer when printed in pytorch shows up as:

(conv): Conv1d(80, 512, kernel_size=(5,), stride=(1,), padding=(2,))

and the parametes in this layer show up as:

postnet.convolutions.0.0.conv.weight : 512x80x5 = 204800

Shouldn't the weights in this layer instead be 512*5*1 as it only has 512 filters each of which is 5x1?

Output:
Intro explanation

Basically Conv1d is just like Conv2d but instead of "sliding" the rectangle window across the image (say 3x3 for kernel_size=3) you "slide" across the vector (say of length 256) with kernel (say of size 3). This is the case for in_channels and out_channels equal to 1 which is the basic one.

Below you can see Conv1d sliding across 3 in_channels (x-axis, y-axis, z-axis) across seconds steps.

You could add depth to the kernel (just like you did for 2D convolution with 5x5x3 cube), which would be 5x3 as well (5 is the kernel size, 3 is the number of in_channels). Now there could be out_channels of those squares (e.g. 56 out_channels) so the final produced sequence is 56 x sequence_length.

Questions

  [...] post-net layer is comprised of 512 filters with shape 5×1" that
  operates on a spectrogram frame 80-d (means 80 different float values
  in the frame), and the result of filter is a 512-d frame.

So your input is 80d (instead of 3 axes like above), kernel_size is the same (5) and out_channels is 512. So the input could look something like this: [64, 80, 256] (for [batch, in_channels, length]) and output would be [64, 512, 256] (provided padding of 3 was used on both sides).

  I don't understand what in_channels, out_channels mean in pytorch
  conv1d definition as in images I can easily understand what
  in-channels/out-channels mean, but for sequence of 80-float values
  frames I'm at loss. What do they mean in the context of seq2seq model
  like this above?

I guess that was answered above. Main point is: the sequence isn't 80-float values! Sequence can be of any length (just like image can be of any size when you pass it to the convolution), here in_channels is 80.

  How do 512, 5x1 filters on 80 float values produce 512 float values?**

512 x sequence_length values are produced on 80 x sequence_length inputs.

  Shouldn't the weights in this layer instead be 512*5*1 as it only has
  512 filters each of which is 5x1?

In PyTorch, in your case, weights would be of shape torch.Size([512, 80, 5]). They could be torch.Size([512, 1, 5]) if you have one input channel, but in this case there are 80 of them.


Instruction: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation?

Input:
I am using pytorch-1.5 to do some gan test. My code is very simple gan code which just fit the sin(x) function:

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# Hyper Parameters
BATCH_SIZE = 64
LR_G = 0.0001
LR_D = 0.0001
N_IDEAS = 5
ART_COMPONENTS = 15
PAINT_POINTS = np.vstack([np.linspace(-1, 1, ART_COMPONENTS) for _ in range(BATCH_SIZE)])

def artist_works():  # painting from the famous artist (real target)
    r = 0.02 * np.random.randn(1, ART_COMPONENTS)
    paintings = np.sin(PAINT_POINTS * np.pi) + r
    paintings = torch.from_numpy(paintings).float()
    return paintings

G = nn.Sequential(  # Generator
    nn.Linear(N_IDEAS, 128),  # random ideas (could from normal distribution)
    nn.ReLU(),
    nn.Linear(128, ART_COMPONENTS),  # making a painting from these random ideas
)

D = nn.Sequential(  # Discriminator
    nn.Linear(ART_COMPONENTS, 128),  # receive art work either from the famous artist or a newbie like G
    nn.ReLU(),
    nn.Linear(128, 1),
    nn.Sigmoid(),  # tell the probability that the art work is made by artist
)

opt_D = torch.optim.Adam(D.parameters(), lr=LR_D)
opt_G = torch.optim.Adam(G.parameters(), lr=LR_G)

for step in range(10000):
    artist_paintings = artist_works()  # real painting from artist
    G_ideas = torch.randn(BATCH_SIZE, N_IDEAS)  # random ideas
    G_paintings = G(G_ideas)  # fake painting from G (random ideas)

    prob_artist0 = D(artist_paintings)  # D try to increase this prob
    prob_artist1 = D(G_paintings)  # D try to reduce this prob

    D_loss = - torch.mean(torch.log(prob_artist0) + torch.log(1. - prob_artist1))
    G_loss = torch.mean(torch.log(1. - prob_artist1))

    opt_D.zero_grad()
    D_loss.backward(retain_graph=True)  # reusing computational graph
    opt_D.step()

    opt_G.zero_grad()
    G_loss.backward()
    opt_G.step()

But when i runing it got this error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Is there something wrong with my code?

Output:
This happens because the opt_D.step() modifies the parameters of your discriminator inplace. But these parameters are required to compute the gradient for the generator. You can fix this by changing your code to:

for step in range(10000):
    artist_paintings = artist_works()  # real painting from artist
    G_ideas = torch.randn(BATCH_SIZE, N_IDEAS)  # random ideas
    G_paintings = G(G_ideas)  # fake painting from G (random ideas)

    prob_artist1 = D(G_paintings)  # G tries to fool D

    G_loss = torch.mean(torch.log(1. - prob_artist1))
    opt_G.zero_grad()
    G_loss.backward()
    opt_G.step()

    prob_artist0 = D(artist_paintings)  # D try to increase this prob
    # detach here to make sure we don't backprop in G that was already changed.
    prob_artist1 = D(G_paintings.detach())  # D try to reduce this prob

    D_loss = - torch.mean(torch.log(prob_artist0) + torch.log(1. - prob_artist1))
    opt_D.zero_grad()
    D_loss.backward(retain_graph=True)  # reusing computational graph
    opt_D.step()

You can find more about this issue here https://github.com/pytorch/pytorch/issues/39141


Instruction: Choice of loss function

Input:
Here is a word2vec implementation:

%reset -f

import torch
from torch.autograd import Variable
import numpy as np
import torch.functional as F
import torch.nn.functional as F

corpus = [
    'this test',
    'this separate test'
]

def get_input_layer(word_idx):
    x = torch.zeros(vocabulary_size).float()
    x[word_idx] = 1.0
    return x

def tokenize_corpus(corpus):
    tokens = [x.split() for x in corpus]
    return tokens

tokenized_corpus = tokenize_corpus(corpus)

vocabulary = []
for sentence in tokenized_corpus:
    for token in sentence:
        if token not in vocabulary:
            vocabulary.append(token)

word2idx = {w: idx for (idx, w) in enumerate(vocabulary)}
idx2word = {idx: w for (idx, w) in enumerate(vocabulary)}

window_size = 2
idx_pairs = []
# for each sentence
for sentence in tokenized_corpus:
    indices = [word2idx[word] for word in sentence]
    # for each word, threated as center word
    for center_word_pos in range(len(indices)):
        # for each window position
        for w in range(-window_size, window_size + 1):
            context_word_pos = center_word_pos + w
            # make soure not jump out sentence
            if context_word_pos < 0 or context_word_pos >= len(indices) or center_word_pos == context_word_pos:
                continue
            context_word_idx = indices[context_word_pos]
            idx_pairs.append((indices[center_word_pos], context_word_idx))

idx_pairs = np.array(idx_pairs) # it will be useful to have this as numpy array

vocabulary_size = len(vocabulary)

embedding_dims = 4
W1 = Variable(torch.randn(embedding_dims, vocabulary_size).float(), requires_grad=True)
W2 = Variable(torch.randn(vocabulary_size, embedding_dims).float(), requires_grad=True)
num_epochs = 1
learning_rate = 0.001

for epo in range(num_epochs):
    loss_val = 0
    for data, target in idx_pairs:
        x = Variable(get_input_layer(data)).float()
        y_true = Variable(torch.from_numpy(np.array([target])).long())

        z1 = torch.matmul(W1, x)
        z2 = torch.matmul(W2, z1)

        log_softmax = F.log_softmax(z2, dim=0)

        loss = F.nll_loss(log_softmax.view(1,-1), y_true)
        print(float(loss))

        loss_val += loss.data.item()
        loss.backward()
        W1.data -= learning_rate * W1.grad.data
        W2.data -= learning_rate * W2.grad.data

        W1.grad.data.zero_()
        W2.grad.data.zero_()

        print(W1.shape)
        print(W2.shape)

        print(f'Loss at epo {epo}: {loss_val/len(idx_pairs)}')

This prints:

0.33185482025146484
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.041481852531433105
3.302438735961914
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.45428669452667236
2.3144636154174805
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.7435946464538574
0.33418864011764526
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.7853682264685631
1.0644199848175049
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.9184207245707512
0.4970806837081909
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.980555810034275
3.2861199378967285
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 1.3913208022713661
6.170125961303711
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 2.16258654743433

Modifying the code to use mse_loss change y_true to float :

y_true = Variable(torch.from_numpy(np.array([target])).float())

Use mse_loss :

loss = F.mse_loss(log_softmax.view(1,-1), y_true)

Incorporated updates:

for epo in range(num_epochs):
    loss_val = 0
    for data, target in idx_pairs:
        x = Variable(get_input_layer(data)).float()
        y_true = Variable(torch.from_numpy(np.array([target])).float())

        z1 = torch.matmul(W1, x)
        z2 = torch.matmul(W2, z1)

        log_softmax = F.log_softmax(z2, dim=0)

        loss = F.mse_loss(log_softmax.view(1,-1), y_true)
        print(float(loss))

        loss_val += loss.data.item()
        loss.backward()
        W1.data -= learning_rate * W1.grad.data
        W2.data -= learning_rate * W2.grad.data

        W1.grad.data.zero_()
        W2.grad.data.zero_()

        print(W1.shape)
        print(W2.shape)

        print(f'Loss at epo {epo}: {loss_val/len(idx_pairs)}')

Output is now:

41.75048828125
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 5.21881103515625
16.929386138916016
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 7.334984302520752
50.63690948486328
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 13.664597988128662
36.21110534667969
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 18.190986156463623
5.304859638214111
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 18.854093611240387
9.802173614501953
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 20.07936531305313
15.515325546264648
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 22.018781006336212
30.408292770385742
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 25.81981760263443

-c:12: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

Why does mse loss not work as well as nll loss ? Is it related to the PyTorch warning:

Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

?

Output:
The input and target must have the same size for the nn.MSELoss, because it is calculated by squaring the difference of the i-th element of the input and the i-th element of the target, i.e. mse_i = (input_i - target_i) ** 2.

Furthermore, your targets are non-negative integers in the range [0, vocabulary_size], but you use log-softmax, which has values in range [-∞, 0]. With MSE the idea is to get the prediction value to the same value as the target, but the only overlap of the two intervals is 0. That means that every class besides 0 is unreachable.

MSE is a regression loss function and is simply not appropriate in this situation, since you are dealing with categorical data.


Instruction: TypeError: forward() takes 1 positional argument but 2 were given

Input:
class DeConv2d(nn.Module):
    def __init__(self, in_channel, out_channel, kernel_size, stride, padding, dilation):
        super().__init__()

        self.up = nn.Upsample(scale_factor=2, mode='nearest')
        self.conv = nn.Conv2d(in_channel, out_channel, kernel_size=kernel_size, \
                              stride=stride, padding=padding, dilation=dilation)

    def forward(self, x):

        output = self.up(x)
        output = self.conv(output)

        return output

class EncoderDecoder(nn.Module):
    def __init__(self, pretrained_net, n_class):
        super().__init__()
        self.n_class = n_class
        self.pretrained_net = pretrained_net
        self.relu = nn.ReLU(inplace=True)

        self.deconv1 = DeConv2d(512, 512, kernel_size=3, stride=1, padding=1, dilation=1)
        self.bn1 = nn.BatchNorm2d(512)

        self.deconv2 = DeConv2d(512, 256, kernel_size=3, stride=1, padding=1, dilation=1)
        self.bn2 = nn.BatchNorm2d(256)

        self.deconv3 = DeConv2d(256, 128, kernel_size=3, stride=1, padding=1, dilation=1)
        self.bn3 = nn.BatchNorm2d(128)

        self.deconv4 = DeConv2d(128, 64, kernel_size=3, stride=1, padding=1, dilation=1)
        self.bn4 = nn.BatchNorm2d(64)

        self.classifier = nn.Conv2d(64, n_class, kernel_size=1)

    def forward(self, x):

        output=self.pretrained_net.layers(x)

        output=self.relu(self.deconv1(output))
        output=self.bn1(output)

        output=self.relu(self.deconv2(output))
        output=self.bn2(output)

        output=self.relu(self.deconv3(output))
        output=self.bn3(output)

        output=self.relu(self.deconv4(output))
        output=self.bn4(output)

        output=self.classifier(output)

        return output

this is my code and I don't know why the type error exist . Does somebody know how to fix these problems?

Output:
When you create a class, and define a function with self args within the class, self is autofilled with the class.Ex:

class item():# create a random class
 self.var = 0
 def fun(self,x):
  self.var +=x
n = item()

And you can try add:

n.fun(3)
print(n.var)

returns 3

self argument is autofilled with class itself


Instruction: Unknown behaviour of HOOKS in PyTorch

Input:
I have a straightforward and simple CNN below,

# creat a dummy deep net
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1,2, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv2 = nn.Conv2d(2,3, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv3 = nn.Conv2d(3,1, kernel_size=3, stride=1, padding=1, bias=True)
        self.seq = nn.Sequential(
                    nn.Conv2d(1,5, kernel_size=3, stride=1, padding=1, bias=True),
                    nn.LeakyReLU(negative_slope=0.2, inplace=True),
                    nn.Conv2d(5,1, kernel_size=3, stride=1, padding=1, bias=True),
                    )
        self.relu = nn.LeakyReLU(negative_slope=0.2, inplace=True)

    def forward(self, x):

        out = self.relu(self.conv1(x))
        out = self.conv3(self.conv2(out))
        out = out + x
        out = self.seq(x)

        return out

5 hooks have been applied to each layer for the forward pass.

Hooked 0 to Conv2d(1, 2, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Hooked 1 to Conv2d(2, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Hooked 2 to Conv2d(3, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Hooked 3 to Sequential(
  (0): Conv2d(1, 5, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): LeakyReLU(negative_slope=0.2, inplace=True)
  (2): Conv2d(5, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
)
Hooked 4 to LeakyReLU(negative_slope=0.2, inplace=True)

These hooks have been created using following class

# ------------------The Hook class begins to calculate each layer stats
class Hook():
    def __init__(self, module, backward=False):
        if backward==False:
            self.hook = module.register_forward_hook(self.hook_fn)
        else:
            self.hook = module.register_backward_hook(self.hook_fn)

        self.inputMean = []
        self.outputMean = []

    def hook_fn(self, module, input, output):
        self.inputMean.append(input[0][0,...].mean().item())#calculate only for 1st image in the batch
        print('\nIn hook class input {}'.format(input[0].size()))
        self.outputMean.append(output[0][0,...].mean().item())
        print('In hook class outout {}'.format(output[0].size()))

# create hooks on each layer
hookF = []
for i,layer in enumerate(list(net.children())):
    print('Hooked to {}'.format(layer))
    hookF.append(Hook(layer))

Please note between Hook 1 and Hook 2 there is no ReLU
self.conv3(self.conv2(out)). Thus OUTPUT of HOOK1 is INPUT to HOOK2 and should be identical. BUT THIS DOES NOT TURNS OUT TO BE WHY? Below is output for HOOK1 and HOOK2

Hook of layer 1 (HOOK on layer 1 which is self.conv2)
... OutputMean: [0.2381615787744522, 0.2710852324962616, 0.30706286430358887, 0.26064932346343994, 0.24395985901355743]

 Hook of layer 2 (HOOK on layer 2 which is self.conv3)
InputMean: [0.13127394020557404, 0.1611362248659134, 0.1457807868719101, 0.17380955815315247, 0.1537724733352661], OutputMean: ...

These two values should have been the same but do not turn out to be.

------ The Full code is shown below -------

import torch
import torch.nn as nn

# creat a dummy deep net
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1,2, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv2 = nn.Conv2d(2,3, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv3 = nn.Conv2d(3,1, kernel_size=3, stride=1, padding=1, bias=True)
        self.seq = nn.Sequential(
                    nn.Conv2d(1,5, kernel_size=3, stride=1, padding=1, bias=True),
                    nn.LeakyReLU(negative_slope=0.2, inplace=True),
                    nn.Conv2d(5,1, kernel_size=3, stride=1, padding=1, bias=True),
                    )
        self.relu = nn.LeakyReLU(negative_slope=0.2, inplace=True)

    def forward(self, x):

        out = self.relu(self.conv1(x))
        out = self.conv3(self.conv2(out))
        out = out + x
        out = self.seq(x)

        return out

net = Net()
print(net)
criterion = nn.MSELoss()

# ------------------The Hook class begins to calculate each layer stats
class Hook():
    def __init__(self, module, backward=False):
        if backward==False:
            self.hook = module.register_forward_hook(self.hook_fn)
        else:
            self.hook = module.register_backward_hook(self.hook_fn)

        self.inputMean = []
        self.outputMean = []

    def hook_fn(self, module, input, output):
        self.inputMean.append(input[0][0,...].mean().item())#calculate only for 1st image in the batch
        print('\nIn hook class input {}'.format(input[0].size()))
        self.outputMean.append(output[0][0,...].mean().item())
        print('In hook class outout {}'.format(output[0].size()))

# create hooks on each layer
hookF = []
for i,layer in enumerate(list(net.children())):
    print('Hooked to {}'.format(layer))
    hookF.append(Hook(layer))

optimizer = torch.optim.Adam(net.parameters())

# Do 5 forward pass
for _ in range(5):
    print('Iteration --------')
    data = torch.rand(2,1,10,10)*10
    print('Input mean is {}'.format(data[0,...].mean()))
    target = data.clone()

    out = net(data)
    loss = criterion(out, target)
    print('backward')
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

for i,h in enumerate(hookF):
    print('\n Hook of layer {}'.format(i))
    print('InputMean: {}, OutputMean: {}'.format(h.inputMean, h.outputMean))
    h.hook.remove()

Output:
The problem is that in your Conv2d layer input is a tuple and output is a torch.Tensor. Therefore output[0][0,...] is selecting the first item from dim 0 in the tensor whereas input[0][0,...] is selecting the first item from the tuple.

You just need to change output[0][0,...] to output[0,...].


Instruction: Understanding accumulated gradients in PyTorch

Input:
I am trying to comprehend inner workings of the gradient accumulation in PyTorch. My question is somewhat related to these two:

Why do we need to call zero_grad() in PyTorch?

Why do we need to explicitly call zero_grad()?

Comments to the accepted answer to the second question suggest that accumulated gradients can be used if a minibatch is too large to perform a gradient update in a single forward pass, and thus has to be split into multiple sub-batches.

Consider the following toy example:

import numpy as np
import torch

class ExampleLinear(torch.nn.Module):

    def __init__(self):
        super().__init__()
        # Initialize the weight at 1
        self.weight = torch.nn.Parameter(torch.Tensor([1]).float(),
                                         requires_grad=True)

    def forward(self, x):
        return self.weight * x

if __name__ == "__main__":
    # Example 1
    model = ExampleLinear()

    # Generate some data
    x = torch.from_numpy(np.array([4, 2])).float()
    y = 2 * x

    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

    y_hat = model(x)          # forward pass

    loss = (y - y_hat) ** 2
    loss = loss.mean()        # MSE loss

    loss.backward()           # backward pass

    optimizer.step()          # weight update

    print(model.weight.grad)  # tensor([-20.])
    print(model.weight)       # tensor([1.2000]

Which is exactly the result one would expect. Now assume that we want to process the dataset sample-by-sample utilizing gradient accumulation:

    # Example 2: MSE sample-by-sample
    model2 = ExampleLinear()
    optimizer = torch.optim.SGD(model2.parameters(), lr=0.01)

    # Compute loss sample-by-sample, then average it over all samples
    loss = []
    for k in range(len(y)):
        y_hat = model2(x[k])
        loss.append((y[k] - y_hat) ** 2)
    loss = sum(loss) / len(y)

    loss.backward()     # backward pass
    optimizer.step()    # weight update

    print(model2.weight.grad)  # tensor([-20.])
    print(model2.weight)       # tensor([1.2000]

Again as expected, the gradient is calculated when the .backward() method is called.

Finally to my question: what exactly happens 'under the hood'?

My understanding is that the computational graph is dynamically updated going from <PowBackward> to <AddBackward> <DivBackward> operations for the loss variable, and that no information about the data used for each forward pass is retained anywhere except for the loss tensor which can be updated until the backward pass.

Are there any caveats to the reasoning in the above paragraph? Lastly, are there any best practices to follow when using gradient accumulation (i.e. can the approach I use in Example 2 backfire somehow)?

Output:
You are not actually accumulating gradients. Just leaving off optimizer.zero_grad() has no effect if you have a single .backward() call, as the gradients are already zero to begin with (technically None but they will be
automatically initialised to zero).
The only difference between your two versions, is how you calculate the final loss. The for loop of the second example does the same calculations as PyTorch does in the first example, but you do them individually, and PyTorch cannot optimise (parallelise and vectorise) your for loop, which makes an especially staggering difference on GPUs, granted that the tensors aren't tiny.
Before getting to gradient accumulation, let's start with your question:

Finally to my question: what exactly happens 'under the hood'?

Every operation on tensors is tracked in a computational graph if and only if one of the operands is already part of a computational graph. When you set requires_grad=True of a tensor, it creates a computational graph with a single vertex, the tensor itself, which will remain a leaf in the graph. Any operation with that tensor will create a new vertex, which is the result of the operation, hence there is an edge from the operands to it, tracking the operation that was performed.
a = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(4.0)
c = a + b # => tensor(6., grad_fn=<AddBackward0>)

a.requires_grad # => True
a.is_leaf # => True

b.requires_grad # => False
b.is_leaf # => True

c.requires_grad # => True
c.is_leaf # => False

Every intermediate tensor automatically requires gradients and has a grad_fn, which is the function to calculate the partial derivatives with respect to its inputs. Thanks to the chain rule, we can traverse the whole graph in reverse order to calculate the derivatives with respect to every single leaf, which are the parameters we want to optimise. That's the idea of backpropagation, also known as reverse mode differentiation. For more details I recommend reading Calculus on Computational Graphs: Backpropagation.
PyTorch uses that exact idea, when you call loss.backward() it traverses the graph in reverse order, starting from loss, and calculates the derivatives for each vertex. Whenever a leaf is reached, the calculated derivative for that tensor is stored in its .grad attribute.
In your first example, that would lead to:
MeanBackward -> PowBackward -> SubBackward -> MulBackward`

The second example is almost identical, except that you calculate the mean manually, and instead of having a single path for the loss, you have multiple paths for each element of the loss calculation. To clarify, the single path also calculates the derivatives of each element, but internally, which again opens up the possibilities for some optimisations.
# Example 1
loss = (y - y_hat) ** 2
# => tensor([16.,  4.], grad_fn=<PowBackward0>)

# Example 2
loss = []
for k in range(len(y)):
    y_hat = model2(x[k])
    loss.append((y[k] - y_hat) ** 2)
loss
# => [tensor([16.], grad_fn=<PowBackward0>), tensor([4.], grad_fn=<PowBackward0>)]

In either case a single graph is created that is backpropagated exactly once, that's the reason it's not considered gradient accumulation.
Gradient Accumulation
Gradient accumulation refers to the situation, where multiple backwards passes are performed before updating the parameters. The goal is to have the same model parameters for multiple inputs (batches) and then update the model's parameters based on all these batches, instead of performing an update after every single batch.
Let's revisit your example. x has size [2], that's the size of our entire dataset. For some reason, we need to calculate the gradients based on the whole dataset. That is naturally the case when using a batch size of 2, since we would have the whole dataset at once. But what happens if we can only have batches of size 1? We could run them individually and update the model after each batch as usual, but then we don't calculate the gradients over the whole dataset.
What we need to do, is run each sample individually with the same model parameters and calculate the gradients without updating the model. Now you might be thinking, isn't that what you did in the second version? Almost, but not quite, and there is a crucial problem in your version, namely that you are using the same amount of memory as in the first version, because you have the same calculations and therefore the same number of values in the computational graph.
How do we free memory? We need to get rid of the tensors of the previous batch and also the computational graph, because that uses a lot of memory to keep track of everything that's necessary for the backpropagation. The computational graph is automatically destroyed when .backward() is called (unless retain_graph=True is specified).
def calculate_loss(x: torch.Tensor) -> torch.Tensor:
    y = 2 * x
    y_hat = model(x)
    loss = (y - y_hat) ** 2
    return loss.mean()

# With mulitple batches of size 1
batches = [torch.tensor([4.0]), torch.tensor([2.0])]

optimizer.zero_grad()
for i, batch in enumerate(batches):
    # The loss needs to be scaled, because the mean should be taken across the whole
    # dataset, which requires the loss to be divided by the number of batches.
    loss = calculate_loss(batch) / len(batches)
    loss.backward()
    print(f"Batch size 1 (batch {i}) - grad: {model.weight.grad}")
    print(f"Batch size 1 (batch {i}) - weight: {model.weight}")

# Updating the model only after all batches
optimizer.step()
print(f"Batch size 1 (final) - grad: {model.weight.grad}")
print(f"Batch size 1 (final) - weight: {model.weight}")

Output (I removed the Parameter containing messages for readability):
Batch size 1 (batch 0) - grad: tensor([-16.])
Batch size 1 (batch 0) - weight: tensor([1.], requires_grad=True)
Batch size 1 (batch 1) - grad: tensor([-20.])
Batch size 1 (batch 1) - weight: tensor([1.], requires_grad=True)
Batch size 1 (final) - grad: tensor([-20.])
Batch size 1 (final) - weight: tensor([1.2000], requires_grad=True)

As you can see, the model kept the same parameter for all batches, while the gradients were accumulate, and there is a single update at the end. Note that the loss needs to be scaled per batch, in order to have the same significance over the whole dataset as if you used a single batch.
While in this example, the whole dataset is used before performing the update, you can easily change that to update the parameters after a certain number of batches, but you have to remember to zero out the gradients after an optimiser step was taken. The general recipe would be:
accumulation_steps = 10
for i, batch in enumerate(batches):
    # Scale the loss to the mean of the accumulated batch size
    loss = calculate_loss(batch) / accumulation_steps
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        # Reset gradients, for the next accumulated batches
        optimizer.zero_grad()

You can find that recipe and more techniques for working with large batch sizes in HuggingFace - Training Neural Nets on Larger Batches: Practical Tips for 1-GPU, Multi-GPU & Distributed setups.


Instruction: IndexError: The shape of the mask [1, 1682] at index 0 does not match the shape of the indexed tensor [100, 1682] at index 0

Input:
I'm currently doing an DeepLearning course in Udemy.
I am currently designing an Restricted Boltzmann machine in which
my training runs perfectly, but I ended up with this error while testing

IndexError: The shape of the mask [1, 1682] at index 0 does not match the shape of the indexed tensor [100, 1682] at index 0

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.optim as optim
import torch.utils.data
from torch.autograd import Variable

# Importing dataset
movies = pd.read_csv('ml-1m/movies.dat',sep='::',header = None,
                     engine='python',encoding='latin-1')
users =  pd.read_csv('ml-1m/users.dat',sep='::',header = None,
                     engine='python',encoding='latin-1')
ratings = pd.read_csv('ml-1m/ratings.dat',sep='::',header = None,
                     engine='python',encoding='latin-1')

# preparing training and test set
training_set = pd.read_csv('ml-100k/u1.base',delimiter='\t')
training_set = np.array(training_set,dtype='int')
test_set = pd.read_csv('ml-100k/u1.test',delimiter='\t')
test_set = np.array(test_set,dtype='int')

#Getting the no of users and movies
nb_users = int(max(max(training_set[:,0]),max(test_set[:,0]))) #max out of both
nb_movies = int(max(max(training_set[:,1]),max(test_set[:,1])))

#Array with users in lines and movies in columns
def convert(data):
    new_data = []
    for id_users in range(1,nb_users+1):
        id_movies = data[:,1][data[:,0]==id_users]
        id_ratings = data[:,2][data[:,0]==id_users]
        ratings = np.zeros(nb_movies)
        ratings[id_movies-1]= id_ratings
        new_data.append(list(ratings))
    return new_data

training_set = convert(training_set)
test_set = convert(test_set)

# COnverting data to torcch tensors
training_set = torch.FloatTensor(training_set)
test_set = torch.FloatTensor(test_set)

# Converting the rating into binary ratings 1 (Liked) or 0 (Not liked)
training_set[training_set == 0] = -1 #taking all zero values in trainingset
training_set[training_set == 1] = 0
training_set[training_set == 2] = 0
training_set[training_set >= 3] = 1
test_set[test_set == 0] = -1 #taking all zero values in trainingset
test_set[test_set == 1] = 0
test_set[test_set == 2] = 0
test_set[test_set >= 3] = 1

# Creating the architecture of the Neural network
class RBM():
    def __init__(self, nv, nh):
        self.W = torch.randn(nh, nv)
        self.a = torch.randn(1, nh)
        self.b = torch.randn(1, nv)
    def sample_h(self, x):
        wx = torch.mm(x, self.W.t())
        activation = wx + self.a.expand_as(wx)
        p_h_given_v = torch.sigmoid(activation)
        return p_h_given_v, torch.bernoulli(p_h_given_v)
    def sample_v(self, y):
        wy = torch.mm(y, self.W)
        activation = wy + self.b.expand_as(wy)
        p_v_given_h = torch.sigmoid(activation)
        return p_v_given_h, torch.bernoulli(p_v_given_h)
    def train(self, v0, vk, ph0, phk):
        self.W += (torch.mm(v0.t(), ph0) - torch.mm(vk.t(), phk)).t()
        self.b += torch.sum((v0 - vk), 0)
        self.a += torch.sum((ph0 - phk), 0)
nv = len(training_set[0])
nh = 100
batch_size = 100
rbm = RBM(nv, nh)

#Training the RBM
nb_epoch = 10
for epoch in range(1, nb_epoch + 1):
    train_loss = 0
    s = 0.
    for id_user in range(0, nb_users - batch_size, batch_size):
        vk = training_set[id_user:id_user+batch_size]
        v0 = training_set[id_user:id_user+batch_size]
        ph0,_ = rbm.sample_h(v0)
        for k in range(10):
            _,hk = rbm.sample_h(vk)
            _,vk = rbm.sample_v(hk)
            vk[v0<0] = v0[v0<0]
        phk,_ = rbm.sample_h(vk)
        rbm.train(v0, vk, ph0, phk)
        train_loss += torch.mean(torch.abs(v0[v0>=0] - vk[v0>=0]))
        s += 1.
    print(f'epoch: {epoch} loss: {train_loss/s}')

# Testing the RBM
test_loss = 0
s = 0.
for id_user in range(nb_users):
    v = training_set[id_user:id_user+1]
    vt = training_set[id_user:id_user+1]
    if len(vt[vt>=0]) > 0:
        _,h = rbm.sample_h(v)
        _,v = rbm.sample_v(hk)
    test_loss += torch.mean(torch.abs(vt[vt>=0] - v[vt>=0]))
    s += 1.
print(f'test_loss: {test_loss/s}')

Output:
In the test loop, you're re-using a variable of the training phase, which I think (you didn't provide the full stacktrace) is just a typo:

for id_user in range(nb_users):
    v = training_set[id_user:id_user+1]
    vt = training_set[id_user:id_user+1]
    if len(vt[vt>=0]) > 0:
        _,h = rbm.sample_h(v)
        _,v = rbm.sample_v(hk)  # hk should be h, shouldn't it?


Instruction: How to do language model training on BERT

Input:
I want to train BERT on a target corpus. I am looking at this HuggingFace implementation.
They are using .raw files for the training data. If I have .txt files of my training data, how can I use their implementation?

Output:
The .raw only indicates that they use the raw version of the WikiText, they are regular text files containing the raw text:

  We're using the raw WikiText-2 (no tokens were replaced before the tokenization).

The description of the data files options also says that they are text files. From run_language_modeling.py - L86-L88:

train_data_file: Optional[str] = field(
    default=None, metadata={"help": "The input training data file (a text file)."}
)

Therefore you can just specify your text files.


Instruction: Does PyTorch have implicit functions for element-wise product and sum?

Input:
I am trying to implement RCAN for super resolution in Tensorflow by Yulun Zhang et al (the original code published with paper is implemented in PyTorch: https://github.com/yulunzhang/RCAN).

I am trying to understand how they have implemented RCAB. By looking at the diagram they have published of their network architecture, it seems pretty straight forward how the nueral network is built. But the code doesn't seem to match it.

According to the diagram here: https://raw.githubusercontent.com/yulunzhang/RCAN/master/Figs/RCAB.PNG

Each RCAB should have following structure:

Residual Channel Attention Block(RCAB){

--0) Conv2D
--1) Relu
--2) Conv2D
--3) Channel Attention Layer{
----0)Global pooling
----1)Conv2D
----2)Relu
----3)Conv2D
----4)Sigmoid
----5)Element Wise Product (Input of this layer/function would be the output from the Conv2D layer 3)
--}
--4) Element Wise Sum (Input of this layer/function would be the input of layer 1)
}

However, when I print the PyTorch model in the paper's GitHub repo, RCAB looks like this:
(see https://github.com/yulunzhang/RCAN/blob/master/RCAN_TrainCode/experiment/model/Network_RCAN_BIX2_G10R20P48-2018-07-15-20-14-55.txt for the full printed model)

(0)RCAB(
          (body): Sequential(
            (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): ReLU(inplace)
            (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (3): CALayer(
              (avg_pool): AdaptiveAvgPool2d(output_size=1)
              (conv_du): Sequential(
                (0): Conv2d(64, 4, kernel_size=(1, 1), stride=(1, 1))
                (1): ReLU(inplace)
                (2): Conv2d(4, 64, kernel_size=(1, 1), stride=(1, 1))
                (3): Sigmoid()
              )
            )
          )
        )

There seems to be no mention of Element Wise sum and Element Wise product in the RCABs of the models that are published along the paper. Signmoid layer is the last layer in each RCAB.

So my question is: does Pytorch have some implicit way of declaring thes element wise sum/product layers? or is it the case that the publishers of the code/model simply haven't added any such layer and therefore have not followed the model architecture diagram that they published?

Output:
If you look at their actual model file you can find the elementwise sum (implemented as just +): https://github.com/yulunzhang/RCAN/blob/master/RCAN_TrainCode/code/model/rcan.py

I believe the elementwise product is handled the same way. These are not exactly "part of the model" in the PyTorch sense. They are not created in __init__ and are kind of dynamic, only revealing its behavior during the forward pass. Static model analysis could not have revealed them (and thus not show in the txt).


Instruction: Setting numpy array to slice without any in-place operations

Input:
How can I do this operation efficiently without any inplace operations?

n_id = np.random.choice(np.arange(2708), size=100)
z = np.random.rand(100, 64)
z_sparse = np.zeros((2708,64))
z_sparse[n_id[:100]] = z

Essentially I want the n_id rows of z_sparse to contain z's rows, but I can't do any inplace operations because my end goal is to use this in a pytorch problem.

One though would be to create zero rows within z precisely so that the rows of z end up in the positions n_id, but not sure how this would work efficiently.

Essentially row 1 of z should be placed at row n_id[0] of z_sparse, then row 2 of z should be at row n_id[1] of z_sparse, and so on...

Here's the PyTorch error jic you are curious:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Output:
If n_id is a fixed index array, you can get z_sparse as a matrix multiplication:

# N, n, m = 2078,100, 64
row_mat = (n_id[:n] == np.arange(N)[:,None])

# for pytorch tensor
# row_mat = Tensor(n_id[:n] == np.arange(N)[:,None])

z_sparse =  row_mat @ z

Since row_mat is a constant array (tensor), your graph should work just fine.


Instruction: How to write a custom loss function in Keras/Tensorflow that uses loops/iterations with reference numpy code

Input:
I saw this question: Implementing custom loss function in keras with condition And I need to do the same thing but with code that seems to need loops.

I have a custom numpy function which calculates the mean Euclid distance from the mean vector. I wrote this based on the paper https://arxiv.org/pdf/1801.05365.pdf:

import numpy as np

def mean_euclid_distance_from_mean_vector(n_vectors):

    dists = []

    for (i, v) in enumerate(n_vectors):
        n_vectors_rest = n_vectors[np.arange(len(n_vectors)) != i]

        print("rest of vectors: ")
        print(n_vectors_rest)

        # calculate mean vector
        mean_rest = n_vectors_rest.mean(axis=0)

        print("mean rest vector")
        print(mean_rest)

        dist = v - mean_rest

        print("dist vector")
        print(dist)
        dists.append(dist)

    # dists is now a matrix of distance vectors (distance from the mean vector)
    dists = np.array(dists)

    print("distance vector matrix")
    print(dists)

    # here we matmult each vector
    # sum them up
    # and divide by the total number of elements
    result = np.sum([np.matmul(d, d) for d in dists]) / dists.size

    return result

features = np.array([
    [1,2,3,4],
    [4,3,2,1]
])

c = mean_euclid_distance_from_mean_vector(features)

print(c)

I need this function however to work inside tensorflow with Keras. So a custom lambda https://www.tensorflow.org/api_docs/python/tf/keras/layers/Lambda

However, I'm not sure how to implement the above in Keras/Tensorflow since it has loops, and the way the paper talked about calculating the m_i seems to require loops like the way I implemented the above.

For reference, the PyTorch version of this code is here: https://github.com/PramuPerera/DeepOneClass

Output:
Given a feature map like:

features = np.array([
    [1, 2, 3, 4],
    [2, 4, 4, 3],
    [3, 2, 1, 4],
], dtype=np.float64)

reflecting a batch_size of

batch_size = features.shape[0]

and

k = features.shape[1]

One has that implementing the above Formulas in Tensorflow could be expressed (prototyped) by:

dim = (batch_size, features.shape[1])
def zero(i):
    arr = np.ones(dim)
    arr[i] = 0
    return arr

mapper = [zero(i) for i in range(batch_size)]
elems = (features, mapper)
m = (1 / (batch_size - 1)) * tf.map_fn(lambda x: tf.math.reduce_sum(x[0] * x[1], axis=0), elems, dtype=tf.float64)
pairs = tf.map_fn(lambda x: tf.concat(x, axis=0) , tf.stack([features, m], 1), dtype=tf.float64)
compactness_loss = (1 / (batch_size * k)) * tf.map_fn(lambda x: tf.math.reduce_euclidean_norm(x), pairs, dtype=tf.float64)

with tf.Session() as sess:
    print("loss value output is: ", compactness_loss.eval())

Which yields:

loss value output is:  [0.64549722 0.79056942 0.64549722]

However a single measure is required for the batch, therefore it is necessary to reduce it; by the summation of all values.

The wanted Compactness Loss function à la Tensorflow is:

def compactness_loss(actual, features):
    features = Flatten()(features)
    k = 7 * 7 * 512
    dim = (batch_size, k)

    def zero(i):
        z = tf.zeros((1, dim[1]), dtype=tf.dtypes.float32)
        o = tf.ones((1, dim[1]), dtype=tf.dtypes.float32)
        arr = []
        for k in range(dim[0]):
            arr.append(o if k != i else z)
        res = tf.concat(arr, axis=0)
        return res

    masks = [zero(i) for i in range(batch_size)]
    m = (1 / (batch_size - 1)) * tf.map_fn(
        # row-wise summation
        lambda mask: tf.math.reduce_sum(features * mask, axis=0),
        masks,
        dtype=tf.float32,
    )
    dists = features - m
    sqrd_dists = tf.pow(dists, 2)
    red_dists = tf.math.reduce_sum(sqrd_dists, axis=1)
    compact_loss = (1 / (batch_size * k)) * tf.math.reduce_sum(red_dists)
    return compact_loss

Of course the Flatten() could be moved back into the model for convenience and the k could be derived directly from the feature map; this answers your question. You may just have some trouble finding out the the expected values for the model are - feature maps from the VGG16 (or any other architechture) trained against the imagenet for instance?

The paper says:

  In  our  formulation  (shown  in  Figure  2  (e)),  starting  froma  pre-trained  deep  model,  we  freeze  initial  features  (gs)  and learn  (gl)  and  (hc).  Based  on  the  output  of  the  classification sub-network (hc),  two  losses compactness  loss and descriptiveness  loss are  evaluated.  These  two  losses,  introduced  in the subsequent sections, are used to assess the quality of the learned  deep  feature.  We  use  the  provided  one-class  dataset to  calculate  the compactness  loss.  An  external  multi-class reference  dataset  is  used  to  evaluate  the descriptiveness  loss.As  shown  in  Figure  3,  weights  of gl and hc are  learned in  the  proposed  method  through  back-propagation  from  the composite loss. Once training is converged, system shown in setup  in  Figure  2(d)  is  used  to  perform  classification  where the resulting model is used as the pre-trained model.

then looking at the "Framework" backbone here plus:

  AlexNet   Binary   and   VGG16   Binary   (Baseline). A  binary  CNN  is  trained  by  having  ImageNet  samples  and  one-class  image  samples  as  the  two  classes  using  AlexNet  andVGG16 architectures, respectively. Testing is performed using k-nearest neighbor, One-class SVM [43], Isolation Forest [3]and Gaussian Mixture Model [3] classifiers.

Makes me wonder whether it would not be reasonable to add suggested the dense layers to both the Secondary and the Reference Networks to a single class output (Sigmoid) or even and binary class output (using Softmax) and using the mean_squared_error as the so called Compactness Loss and binary_cross_entropy as the Descriptveness Loss.


Instruction: Using SVM with different kernels as a last layer in CNN network

Input:
I'm trying to replace the last fully connected layer of a CNN network with SVM using pytorch in a multi-classification problem. I've done some research and it says, that I should just replace the nn.CrossEntropyLoss with nn.MultiMarginLoss.

How does changing the criterion only actually corresponds with the "replacing fully connected layer with SVM" task? Another thing is that I'd like to use the SVM with different kernel, like for example the quadratic one.

Output:
This question can actually be interpreted as the difference between Logistic regression and SVM in classification.

We can naively look at the whole platform of your deep learning as if you have a magician, and that magician accepts the input data, and give you a set of engineered featured, and you use those features to do the classification.

Depending on which loss you minimize, you can solve this classification issue with different sorts of functions. If you use cross-entropy, it is like you are applying a logistic regression classification. On the other hand, if you minimize the marginal loss, it is actually equal to finding the support vectors, which is indeed how SVM works.

You need to read about the role of kernels in the calculation of the loss(for ex, here ), but TL;DR is that for loss computation, you have a component of K(xi,xj) which is actually the kernel function and indicate the similarity of xi and xj.

So you can implement a custom loss, where you have a polynomial kernel (quadratic in your case), and imitate the margin loss calculation there.


Instruction: Preferred way to decrease learning rate for Adam optimiser in PyTorch

Input:
I have been seeing code that uses an Adam optimizer . And the way they decrease the learning rate is as follows:

    optimizer = torch.optim.Adam(net.parameters(),lr=0.01)
    (training...
    optimizer.step()...)

    if iteration >= some_threshold:
        for param_group in optimizer.param_groups:
            param_group['lr'] = 0.001

I thought we have the same learning rate for all parameters. So why then iterate over the param_groups and individually set learning rate for each parameter?

Wouldn't the following be faster and have an identical effect?

    optimizer = torch.optim.Adam(net.parameters(),lr=0.01)
    scheduler = MultiStepLR(optimizer, milestones=[some_threshold], gamma=0.1)
    (training...
    optimizer.step()
    scheduler.step())

Thank you

Output:
You need to iterate over param_groups because if you don't specify multiple groups of parameters in the optimiser, you automatically have a single group. That doesn't mean you set the learning rate for each parameter, but rather each parameter group.

In fact the learning rate schedulers from PyTorch do the same thing. From _LRScheduler (base class of learning rate schedulers):

with _enable_get_lr_call(self):
    if epoch is None:
        self.last_epoch += 1
        values = self.get_lr()
    else:
        warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
        self.last_epoch = epoch
        if hasattr(self, "_get_closed_form_lr"):
            values = self._get_closed_form_lr()
        else:
            values = self.get_lr()

for param_group, lr in zip(self.optimizer.param_groups, values):
    param_group['lr'] = lr

Yes, it has the identical effect in this case, but it wouldn't be faster.


Instruction: How to get embedding from bert finetuned model?

Input:
I have finedtuned 'bert-base-uncased' model using transformer and torch which gave me pytorch_model.bin, vocab.txt and other files as output.
After loading the model how to I get embedding for complete vocab, like a matrix which maps every word to its embedding vector

Output:
I recommend to use Huggingface, they make it very easy to use and train all Transformer model variants.

To get the embedding from a BERT fine-tuned model, you could use BertModel.set_input_embeddings().


Instruction: RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False

Input:
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
I am getting the above error for code:-
    def get_model(path, device):
        model = models.vgg16(pretrained=False)

        for param in model.parameters():
            param.requires_grad = False
        n_inputs = model.classifier[6].in_features

        model.classifier[6] = torch.nn.Sequential(
            torch.nn.Linear(n_inputs, 256), torch.nn.ReLU(), torch.nn.Dropout(0.2),
            torch.nn.Linear(256, 10), torch.nn.LogSoftmax(dim=1))

        model.load_state_dict(torch.load(path), map_location=torch.device('cpu'))
        model.to(device)
        model.eval()
        return model

    device = torch.device("cpu")
    model = get_model('vgg16.pt', device)

Output:
You are passing the map_location to the wrong function (to model.load_state_dict instead of torch.load).

The corrected line would look like this:

model.load_state_dict(torch.load(path, map_location=torch.device('cpu')))


Instruction: RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False, Dataloader Error, and setting pin_memory=False

Input:
I am a beginner trying to evaluate this video object segmentation network paper.

When following the instructions on https://github.com/seoungwugoh/STM

It says the requirements are as follows:-

python 3.6
pytorch 1.0.1.post2
numpy, opencv, pillow

I couldn't get this pytorch version to install, so I installed the conda-forge pytorch version 1.5.

and I run this command in either Windows 10 or Ubuntu 16.04 using Anaconda

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ python eval_DAVIS.py -g '1' -s val -y 16 -D ../DAVISSemiSupervisedTrainVal480

after doing pip install matplotlib, and pip install tqdm ...

I get the following error message:-

Space-time Memory Networks: initialized.
STM : Testing on DAVIS
Loading weights: STM_weights.pth
Traceback (most recent call last):

File "eval_DAVIS.py", line 111, in
  model.load_state_dict(torch.load(pth_path))

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 593, in load
  return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 773, in _legacy_load
  result = unpickler.load()

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 729, in persistent_load

deserialized_objects[root_key] = restore_location(obj, location)

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 178, in default_restore_location
  result = fn(storage, location)

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 154, in _cuda_deserialize
    device = validate_cuda_device(location)

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 138, in validate_cuda_device raise RuntimeError('Attempting to deserialize object on a CUDA '

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU

My Graphics Card Driver, and System and Packages are as follows:-

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0  On |                  N/A |
| 26%   34C    P8    10W / 151W |    392MiB /  8118MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1247      G   /usr/lib/xorg/Xorg                           229MiB |
|    0      2239      G   compiz                                       126MiB |
|    0      9385      G   /usr/lib/firefox/firefox                       2MiB |
|    0     11686      G   /proc/self/exe                                30MiB |
+-----------------------------------------------------------------------------+

I also tried this

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ python -c 'import torch; print(torch.rand(2,3).cuda())'

tensor([[0.9178, 0.8239, 0.4761],
        [0.9429, 0.8877, 0.0097]], device='cuda:0')

Which shows that cuda is working here

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ conda info
    active environment : STMVOS
    active env location : /home/oneworld/anaconda3/envs/STMVOS
            shell level : 1
       user config file : /home/oneworld/.condarc
 populated config files :
          conda version : 4.8.2
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __cuda=10.2
                          __glibc=2.23
       base environment : /home/oneworld/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/oneworld/anaconda3/pkgs
                          /home/oneworld/.conda/pkgs
       envs directories : /home/oneworld/anaconda3/envs
                          /home/oneworld/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Linux/4.4.0-179-generic ubuntu/16.04.6 glibc/2.23
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ conda list

packages in environment at /home/oneworld/anaconda3/envs/STMVOS:

Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
blas                      1.0                         mkl
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
cairo                     1.16.0            hcf35c78_1003    conda-forge
certifi                   2020.4.5.1               py38_0
cudatoolkit               10.2.89              hfd86e86_1
cycler                    0.10.0                   pypi_0    pypi
dbus                      1.13.6               he372182_0    conda-forge
expat                     2.2.9                he1b5a44_2    conda-forge
ffmpeg                    4.2.3                h167e202_0    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.9.1                h8a8886c_1
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
giflib                    5.2.1                h516909a_2    conda-forge
glib                      2.64.3               h6f030ca_0    conda-forge
gmp                       6.2.0                he1b5a44_2    conda-forge
gnutls                    3.6.5             hd3a4fd2_1002    conda-forge
graphite2                 1.3.13            he1b5a44_1001    conda-forge
gst-plugins-base          1.14.5               h0935bb2_2    conda-forge
gstreamer                 1.14.5               h36ae1b5_2    conda-forge
harfbuzz                  2.4.0                h9f30f68_3    conda-forge
hdf5                      1.10.6          nompi_h3c11f04_100    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
intel-openmp              2020.1                      217
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
kiwisolver                1.2.0                    pypi_0    pypi
lame                      3.100             h14c3975_1001    conda-forge
ld_impl_linux-64          2.33.1               h53a641e_7
libblas                   3.8.0                    15_mkl    conda-forge
libcblas                  3.8.0                    15_mkl    conda-forge
libclang                  9.0.1           default_hde54327_0    conda-forge
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libiconv                  1.15              h516909a_1006    conda-forge
liblapack                 3.8.0                    15_mkl    conda-forge
liblapacke                3.8.0                    15_mkl    conda-forge
libllvm9                  9.0.1                he513fc3_1    conda-forge
libopencv                 4.2.0                    py38_6    conda-forge
libpng                    1.6.37               hbc83047_0
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.1.0                h2733197_0
libuuid                   2.32.1            h14c3975_1000    conda-forge
libwebp                   1.0.2                h56121f0_5    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxkbcommon              0.10.0               he1b5a44_0    conda-forge
libxml2                   2.9.10               hee79883_0    conda-forge
matplotlib                3.2.1                    pypi_0    pypi
mkl                       2020.1                      217
mkl-service               2.3.0            py38he904b0f_0
mkl_fft                   1.0.15           py38ha843d7b_0
mkl_random                1.1.1            py38h0573a6f_0
ncurses                   6.2                  he6710b0_1
nettle                    3.4.1             h1bed415_1002    conda-forge
ninja                     1.9.0            py38hfd86e86_0
nspr                      4.25                 he1b5a44_0    conda-forge
nss                       3.47                 he751ad9_0    conda-forge
numpy                     1.18.1           py38h4f9e942_0
numpy-base                1.18.1           py38hde5b4d6_1
olefile                   0.46                       py_0
opencv                    4.2.0                    py38_6    conda-forge
openh264                  2.1.1                h8b12597_0    conda-forge
openssl                   1.1.1g               h516909a_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pillow                    7.1.2            py38hb39fc2d_0
pip                       20.0.2                   py38_3
pixman                    0.38.0            h516909a_1003    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
py-opencv                 4.2.0            py38h23f93f0_6    conda-forge
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.8.1                h0371630_1
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.8                      1_cp38    conda-forge
pytorch                   1.5.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
qt                        5.12.5               hd8c4c69_1    conda-forge
readline                  7.0                  h7b6447c_5
setuptools                46.4.0                   py38_0
six                       1.14.0                   py38_0
sqlite                    3.31.1               h62c20be_1
tk                        8.6.8                hbc83047_0
torchvision               0.6.0                py38_cu102    pytorch
tqdm                      4.46.0                   pypi_0    pypi
wheel                     0.34.2                   py38_0
x264                      1!152.20180806       h14c3975_0    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.9                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.5                h7b6447c_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.3.7                h0b5b093_0

The code it gets stuck on in eval_DAVIS.py is as follows:-

print('Loading weights:', pth_path)
model.load_state_dict(torch.load(pth_path))

I am using Ubuntu 16.04, however I tried a similar setup in windows 10 and received the same error messages.

Any help much appreciated.

Kind regards

OneWorld

Output:
because of the Python error suggestion

if __name__ == '__main__':
    freeze_support()

I added this line

if __name__ == '__main__':

above the line

for seq, V in enumerate(Testloader):

and indented that line and everything else below.

It then worked as far as to the end of [bike packing]

However requested a scipy install before [black swan]

So I did conda install scipy

and reran, and it started to go through the rest [bmx-trees], [breakdance] etc.

The resulting eval_DAVIS.py file looked like this...

from __future__ import division
import torch
from torch.autograd import Variable
from torch.utils import data

import torch.nn as nn
import torch.nn.functional as F
import torch.nn.init as init
import torch.utils.model_zoo as model_zoo
from torchvision import models

# general libs
import cv2
import matplotlib.pyplot as plt
from PIL import Image
import numpy as np
import math
import time
import tqdm
import os
import argparse
import copy

### My libs
from dataset import DAVIS_MO_Test
from model import STM

torch.set_grad_enabled(False) # Volatile

# def get_arguments():
#     parser = argparse.ArgumentParser(description="SST")
#     parser.add_argument("-g", type=str, help="0; 0,1; 0,3; etc", required=True)
#     parser.add_argument("-s", type=str, help="set", required=True)
#     parser.add_argument("-y", type=int, help="year", required=True)
#     parser.add_argument("-viz", help="Save visualization", action="store_true")
#     parser.add_argument("-D", type=str, help="path to data",default='/local/DATA')
#     return parser.parse_args()

# args = get_arguments()

# GPU = args.g
# YEAR = args.y
# SET = args.s
# VIZ = args.viz
# DATA_ROOT = args.D

GPU = '0'
YEAR = '17'
SET = 'val'
VIZ = 'store_true'
DATA_ROOT = '..\\DAVIS2017SemiSupervisedTrainVal480'

# Model and version
MODEL = 'STM'
print(MODEL, ': Testing on DAVIS')

os.environ['CUDA_VISIBLE_DEVICES'] = GPU
if torch.cuda.is_available():
    print('using Cuda devices, num:', torch.cuda.device_count())

if VIZ:
    print('--- Produce mask overaid video outputs. Evaluation will run slow.')
    print('--- Require FFMPEG for encoding, Check folder ./viz')

palette = Image.open(DATA_ROOT + '/Annotations/480p/blackswan/00000.png').getpalette()

def Run_video(Fs, Ms, num_frames, num_objects, Mem_every=None, Mem_number=None):
    # initialize storage tensors
    if Mem_every:
        to_memorize = [int(i) for i in np.arange(0, num_frames, step=Mem_every)]
    elif Mem_number:
        to_memorize = [int(round(i)) for i in np.linspace(0, num_frames, num=Mem_number+2)[:-1]]
    else:
        raise NotImplementedError

    Es = torch.zeros_like(Ms)
    Es[:,:,0] = Ms[:,:,0]

    for t in tqdm.tqdm(range(1, num_frames)):
        # memorize
        with torch.no_grad():
            prev_key, prev_value = model(Fs[:,:,t-1], Es[:,:,t-1], torch.tensor([num_objects]))

        if t-1 == 0: #
            this_keys, this_values = prev_key, prev_value # only prev memory
        else:
            this_keys = torch.cat([keys, prev_key], dim=3)
            this_values = torch.cat([values, prev_value], dim=3)

        # segment
        with torch.no_grad():
            logit = model(Fs[:,:,t], this_keys, this_values, torch.tensor([num_objects]))
        Es[:,:,t] = F.softmax(logit, dim=1)

        # update
        if t-1 in to_memorize:
            keys, values = this_keys, this_values

    pred = np.argmax(Es[0].cpu().numpy(), axis=0).astype(np.uint8)
    return pred, Es

Testset = DAVIS_MO_Test(DATA_ROOT, resolution='480p', imset='20{}/{}.txt'.format(YEAR,SET), single_object=(YEAR==16))
Testloader = data.DataLoader(Testset, batch_size=1, shuffle=False, num_workers=2, pin_memory=True)

model = nn.DataParallel(STM())
if torch.cuda.is_available():
    model.cuda()
model.eval() # turn-off BN

pth_path = 'STM_weights.pth'
print('Loading weights:', pth_path)
model.load_state_dict(torch.load(pth_path)) # , map_location=torch.device('cpu')

code_name = '{}_DAVIS_{}{}'.format(MODEL,YEAR,SET)
print('Start Testing:', code_name)

if torch.cuda.is_available() == False:
    print("********** CUDA is NOT available just before line of error **********")
else:
    print("********** CUDA is available, and working fine just before line of error ***********")

if __name__ == '__main__':

    for seq, V in enumerate(Testloader):
        Fs, Ms, num_objects, info = V
        seq_name = info['name'][0]
        num_frames = info['num_frames'][0].item()
        print('[{}]: num_frames: {}, num_objects: {}'.format(seq_name, num_frames, num_objects[0][0]))

        pred, Es = Run_video(Fs, Ms, num_frames, num_objects, Mem_every=5, Mem_number=None)

        # Save results for quantitative eval ######################
        test_path = os.path.join('./test', code_name, seq_name)
        if not os.path.exists(test_path):
            os.makedirs(test_path)
        for f in range(num_frames):
            img_E = Image.fromarray(pred[f])
            img_E.putpalette(palette)
            img_E.save(os.path.join(test_path, '{:05d}.png'.format(f)))

        if VIZ:
            from helpers import overlay_davis
            # visualize results #######################
            viz_path = os.path.join('./viz/', code_name, seq_name)
            if not os.path.exists(viz_path):
                os.makedirs(viz_path)

            for f in range(num_frames):
                pF = (Fs[0,:,f].permute(1,2,0).numpy() * 255.).astype(np.uint8)
                pE = pred[f]
                canvas = overlay_davis(pF, pE, palette)
                canvas = Image.fromarray(canvas)
                canvas.save(os.path.join(viz_path, 'f{}.jpg'.format(f)))

            vid_path = os.path.join('./viz/', code_name, '{}.mp4'.format(seq_name))
            frame_path = os.path.join('./viz/', code_name, seq_name, 'f%d.jpg')
            os.system('ffmpeg -framerate 10 -i {} {} -vcodec libx264 -crf 10  -pix_fmt yuv420p  -nostats -loglevel 0 -y'.format(frame_path, vid_path))

However...

Eventually I got an out of memory error

[car-shadow]: num_frames: 40, num_objects: 1
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [00:09<00:00,  3.98it/s]
Traceback (most recent call last):
  File "eval_DAVIS.py", line 129, in <module>
    for seq, V in enumerate(Testloader):
  File "C:\Users\OneWorld\anaconda3\envs\STMVOS\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\OneWorld\anaconda3\envs\STMVOS\lib\site-packages\torch\utils\data\dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "C:\Users\OneWorld\anaconda3\envs\STMVOS\lib\site-packages\torch\utils\data\dataloader.py", line 881, in _process_data
    data.reraise()
  File "C:\Users\OneWorld\anaconda3\envs\STMVOS\lib\site-packages\torch\_utils.py", line 395, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in pin memory thread for device 0.
Original Traceback (most recent call last):
  File "C:\Users\OneWorld\anaconda3\envs\STMVOS\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line 31, in _pin_memory_loop
    data = pin_memory(data)
  File "C:\Users\OneWorld\anaconda3\envs\STMVOS\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line 55, in pin_memory
    return [pin_memory(sample) for sample in data]
  File "C:\Users\OneWorld\anaconda3\envs\STMVOS\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line 55, in <listcomp>
    return [pin_memory(sample) for sample in data]
  File "C:\Users\OneWorld\anaconda3\envs\STMVOS\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line 47, in pin_memory
    return data.pin_memory()
RuntimeError: cuda runtime error (2) : out of memory at ..\aten\src\THC\THCCachingHostAllocator.cpp:278

so I set the testloader from pin_memory=True to false at around line 108 in eval_DAVIS.py

Testloader = data.DataLoader(Testset, batch_size=1, shuffle=False, num_workers=2, pin_memory=False)

and reran.

Seemed to work fine.


Instruction: PyTorch model saving error: "Can't pickle local object"

Input:
When I try to save the PyTorch model with this piece of code:
checkpoint = {'model': Net(), 'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

I get the following error:
    E:\PROGRAM FILES\Anaconda\envs\staj_projesi\lib\site-packages\torch\serialization.py:251: UserWarning: Couldn't retrieve source code for container of type Net. It won't be checked for correctness upon loading.
...

      "type " + obj.__name__ + ". It won't be checked "
    Can't pickle local object 'trainModel.<locals>.Net'

When I try to save the PyTorch model with this piece of code:
checkpoint = {'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

I don't don't get any errors, but I want to save the ANN class. How can I solve this problem? Also, I could save the model with the first structure in the other projects before

Output:
You can't! torch.save is saving the objects state_dict() only.

When you use the following:

checkpoint = {'model': Net(), 'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

You are trying to save the model itself, but this data is saved in the model.state_dict() and when loading a model with the state_dict you should first initiate a model object.

This is exactly the reason why the second method works properly:

checkpoint = {'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

I would suggest reading the pytorch docs of how to properly save\load a model in the following link:
https://pytorch.org/tutorials/beginner/saving_loading_models.html


Instruction: Having a network output as another network parameters

Input:
I have y = Y(x;theta) and theta = M(t;omega), where x and t are input variables (given from the dataset), and theta and omega trainable parameters. I need to have theta as function of omega. Then, I have a loss function over y and need to backpropagate the gradient through M up to Y. How can I create such a structure in pytorch?

Currently, my network is built as follows (sizes is a list of integers, defined as sizes = [input_size, hidden1_size, hidden2_size, ..., output_size])

import torch
import torch.nn as nn
import torch.nn.functional as F

class M(nn.Module):
    def __init__(self, sizes):
        super(Y, self).__init__()
        self.layers = nn.ModuleList()
        for i in range(0, len(sizes) - 1):
            self.layers.append(nn.Linear(sizes[i], sizes[i+1]))

    def forward(self, x):
        for l in self.layers[:-1]:
            x = F.relu(l(x))
        x = self.layers[-1](x)

        return x

Output:
I think it is quite simple or I didn't get your query correctly.

x, t are your input variables.

Now let us define a network M that will take input t and output theta.

M = nn.Sequential(....) # declare network here

Next, we define a network Y. This here might be tricky as you want to use theta as parameters. It might be easier and intuitive to work with functional counterparts of the modules declared in nn (see https://pytorch.org/docs/stable/nn.functional.html). I will try to give an example of this assuming theta are params of a linear module.

class Y(nn.Module):
    def __init__(self):
        # declare any modules here

    def forward(self, theta, x):
        return nn.functional.linear(input=x, weight=theta, bias=None)

The overall forward pass would be

def forward(t, x, M, Y):
    theta = M(t)
    output = Y(theta, x)
    return output


Instruction: why doesnt Stochastic gradient descent fluctuate

Input:
In batch gradient descent the parameters were updated based on the total/average loss of all the points
In Stochastic gradient descent or SGD
we are updating the parameters after every point instead of one epoch.
so lets say if the final point is an outlier woudnt that cause the whole fitted line to fluctuate drastically.
How is it reliable .
or converge on a contour like this SGD contour

Output:
While it is true that in its most pristine form SGD operates on just 1 sample point, in reality this is not the dominant practice. In practice, we use a mini-batch of say 256, 128 or 64 samples rather than operating on the full batch size containing all the samples in the database, which might be well over than 1 million samples. So clearly operating on a mini-batch of say 256 is much faster than operating on 1 million points and at the same time helps curb the variability caused due to just using 1 sample point.
A second point is that there is no final point. One simply keeps iterating over the dataset. The learning rate for SGD is generally quite small say 1e-3. So even if a sample point happens to be an outlier, the wrong gradients will be scaled by 1e-3 and hence SGD will not be too much off the correct trajectory. When it iterates over the upcoming sample points, which are not outliers, it will again head towards the correct direction.

So altogether using a medium-sized mini-batch and using a small learning rate helps SGD to not digress a lot from the correct trajectory.

Now the word stochastic in SGD can also imply various other measures. For example some practitioners also use gradient clipping i.e. they clamp the calculated gradient to maximum value if the gradients are well over this decided maximum threshold. You can find more on gradient clipping in this post. Now, this is just one trick amongst dozens of other techniques and if you are interested can read source code of popular implementation of SGD in PyTorch or TensorFlow.


Instruction: How to set random seed when it is in distributed training in PyTorch?

Input:
Now I am training a model using torch.distributed, but I am not sure how to set the random seeds. For example, this is my current code:

def main():
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.cuda.manual_seed(args.seed)

    cudnn.enabled = True
    cudnn.benchmark = True
    cudnn.deterministic = True

    mp.spawn(main_worker, nprocs=args.ngpus, args=(args,))

And should I move the

    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.cuda.manual_seed(args.seed)

    cudnn.enabled = True
    cudnn.benchmark = True
    cudnn.deterministic = True

into the function main_worker() to make sure every process has the correct seed and cudnn settings? By the way, I have tried this, and this behavior will make the training 2 times slower, which really confused me.

Thank you very much for any help!

Output:
The spawned child processes do not inherit the seed you set manually in the parent process, therefore you need to set the seed in the main_worker function.

The same logic applies to cudnn.benchmark and cudnn.deterministic, so if you want to use these, you have to set them in main_worker as well. If you want to verify that, you can just print their values in each process.

cudnn.benchmark = True tries to find the optimal algorithm for your model, by benchmarking various implementations of certain operations (e.g. available convolution algorithms). This will take time to find the best algorithm, but once that is done, further iterations will potentially be faster. The algorithm that was determined to be the best, only applies to the specific input size that was used. If in the next iteration you have a different input size, the benchmark needs to be run again, in order to determine the best algorithm for that specific input size, which might be a different one than for the first input size.

I'm assuming that your input sizes vary, which would explain the slow down, as the benchmark wasn't used when it was set in the parent process. cudnn.benchmark = True should only be used if your input sizes are fixed.

cudnn.determinstic = True may also have a negative impact on the performance, because certain underlying operations, that are non-deterministic, need to be replaced with a deterministic version, which tend to be slower, otherwise the deterministic version would be used in the first place, but that performance impact shouldn't be too dramatic.


Instruction: Need help understanding the gradient function in pytorch

Input:
The following code

w = np.array([[2., 2.],[2., 2.]])
x = np.array([[3., 3.],[3., 3.]])
b = np.array([[4., 4.],[4., 4.]])
w = torch.tensor(w, requires_grad=True)
x = torch.tensor(x, requires_grad=True)
b = torch.tensor(b, requires_grad=True)

y = w*x + b
print(y)
# tensor([[10., 10.],
#         [10., 10.]], dtype=torch.float64, grad_fn=<AddBackward0>)

y.backward(torch.FloatTensor([[1, 1],[ 1, 1]]))

print(w.grad)
# tensor([[3., 3.],
#         [3., 3.]], dtype=torch.float64)

print(x.grad)
# tensor([[2., 2.],
#         [2., 2.]], dtype=torch.float64)

print(b.grad)
# tensor([[1., 1.],
#         [1., 1.]], dtype=torch.float64)

As the tensor argument inside gradient function is an all ones tensor in the shape of the input tensor, my understanding says that

w.grad means derivative of y w.r.t w, and produces b,
x.grad means derivative of y w.r.t x, and produces b and
b.grad means derivative of y w.r.t b, and produces all ones.

Out of these, only point 3 answer is matching my expected result. Can someone help me in understanding the first two answers. I think I understand the accumulation part, but don't think that is happening here.

Output:
To find the correct derivatives in this example, we need to take the sum and product rule into consideration.

Sum rule:

Product rule:

That means the derivatives of your equation are calculated as follows.

With respect to x:

With respect to w:

With respect to b:

The gradients reflect exactly that:

torch.equal(w.grad, x) # => True

torch.equal(x.grad, w) # => True

torch.equal(b.grad, torch.tensor([[1, 1], [1, 1]], dtype=torch.float64)) # => True