Instruction: Explain CUDA out of memory in Pytorch

Input:
Can anybody help me to explain the meaning of this common problem in Pytorch?
Model: EfficientDet-D4
GPU: RTX 2080Ti
Batch size: 2
CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 11.00 GiB total capacity; 8.32 GiB already allocated; 2.59 MiB free; 8.37 GiB reserved in total by PyTorch)

Anyway, I think the model and GPU are not important here and I know the solution should be reduced batch size, try to turn off the gradient while validating, etc. But I just want to know what is the meaning of 8.32 GiB while I have 11 GiB but can not allocate 14.00 MiB more?
Addition: I try to watch nvidia-smi while training with batch size = 1, it took 9.5 GiB in my GPU.

Output:
I have the answer from @ptrblck in the Pytorch community. In there, I described my question in more detail than this question.
Please check the answer in here .


Instruction: How to convert a PyTorch sparse_coo_tensor into a PyTorch dense tensor?

Input:
I create a sparse_coo tensor in PyTorch:
import torch

# create indices
i = torch.tensor([[0, 1, 1],
                  [2, 0, 2]])
# create values
v = torch.tensor([3, 4, 5], dtype=torch.float32)

# create sparse_coo_tensor
sparse_tensor = torch.sparse_coo_tensor(i, v, [2, 4])

Now I want to convert a PyTorch sparse tensor into a PyTorch dense tensor. Which function can be used?

Output:
you can use to_dense as suggested in this example :
s = torch.sparse_coo_tensor(i, v, [2, 4])
s_dense = s.to_dense()

And by the way, the documentation is here


Instruction: Train network coded with nn.ModuleDict

Input:
I was assigned to write a simple network with nn.ModuleDict. So, here it is:
third_model = torch.nn.ModuleDict({
'flatten': torch.nn.Flatten(),

'fc1': torch.nn.Linear(32 * 32 * 3, 1024),
'relu': torch.nn.ReLU(),

'fc2': torch.nn.Linear(1024, 240),
'relu': torch.nn.ReLU(),

'fc3': torch.nn.Linear(240, 10)})

Then I tried to train it (with cuda):
third_model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(third_model.parameters(), lr=0.001, momentum=0.9)

train(third_model, criterion, optimizer, train_dataloader, test_dataloader)

Function "train(model, criterion, optimizer, train_dataloader, test_dataloader)" trains the model and visualizes loss and accuracy of the model. It works properly.
Train:
def train(model, criterion, optimizer, train_dataloader, test_dataloader):
    train_loss_log = []
    train_acc_log = []
    val_loss_log = []
    val_acc_log = []

    for epoch in range(NUM_EPOCH):
        model.train()
        train_loss = 0.
        train_size = 0
        train_acc = 0.
        for imgs, labels in train_dataloader:
            imgs, labels = imgs.to(device), labels.to(device)

            optimizer.zero_grad()
            y_pred = model(imgs)
            loss = criterion(y_pred, labels)
            loss.backward()
            optimizer.step()

            train_loss += loss.item()
            train_size += y_pred.size(0)
            train_loss_log.append(loss.data / y_pred.size(0))

            _, pred_classes = torch.max(y_pred, 1)
            train_acc += (pred_classes == labels).sum().item()
            train_acc_log.append(np.mean((pred_classes == labels).cpu().numpy()))

        val_loss = 0.
        val_size = 0
        val_acc = 0.
        model.eval()
        with torch.no_grad():
            for imgs, labels in test_dataloader:
                imgs, labels = imgs.to(device), labels.to(device)
                pred = model(imgs)
                loss = criterion(pred, labels)
                val_loss += loss.item()
                val_size += pred.size(0)

                _, pred_classes = torch.max(pred, 1)
                val_acc += (pred_classes == labels).sum().item()
        val_loss_log.append(val_loss / val_size)
        val_acc_log.append(val_acc / val_size)

        clear_output()
        plot_history(train_loss_log, val_loss_log, 'loss')
        plot_history(train_acc_log, val_acc_log, 'accuracy')

        print('Train loss:', train_loss / train_size)
        print('Train acc:', train_acc / train_size)
        print('Val loss:', val_loss / val_size)
        print('Val acc:', val_acc / val_size)

I've already trained models coded with nn.Sequential and everything is okay. However, with nn.ModuleDict I get an error:
TypeError                                 Traceback (most recent call last)
<ipython-input-144-8b33ad3aad2c> in <module>()
      2 optimizer = optim.SGD(third_model.parameters(), lr=0.001, momentum=0.9)
      3
----> 4 train(third_model, criterion, optimizer, train_dataloader, test_dataloader)

1 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

TypeError: forward() takes 1 positional argument but 2 were given

Tried to find any documentation on nn.ModuleDict, but it seems like there are no examples of coding networks with it.
It seems like the problem might be with linear layers, although I do not know why.
So, I hope anyone could explain where the mistake is. Would be very grateful for any possible advice.

Output:
A nn.moduleDict is a container and its forward function is not defined. It should be used to store sub-modules/networks.
You should using nn.Sequential initialized with as ordered dictionary, OrderedDict:
third_model = torch.nn.Sequential(
    OrderedDict([
       ('flatten', torch.nn.Flatten()),
       ('fc1', torch.nn.Linear(32 * 32 * 3, 1024)),
       ('relu', torch.nn.ReLU()),
       ('fc2', torch.nn.Linear(1024, 240)),
       ('relu', torch.nn.ReLU()),
       ('fc3', torch.nn.Linear(240, 10))]))


Instruction: Why loaded Pytorch model's loss highly increased?

Input:
I'm trying to train Arcface with reference to.
As far as I know, Arcface requires more than 200 training epochs on CASIA-webface with a large batch size.
Within 100 epochs of training, I stopped the training for a while because I was needed to use GPU for other tasks. And the checkpoints of the model(Resnet) and margin are saved. Before it was stopped, its loss recorded a value between 0.3~1.0, and training accuracy was mount to 80~95%.
When I resume the Arcface training by loading the checkpoint files using load_sate, it seems normal when the first batch is processed. But suddenly the loss increased sharply and the accuracy became very low.
Its loss suddenly became increased. How did this happen? I had no other way so anyway continued the training, but I don't think the loss is decreasing well even though it is a trained model over 100 epochs...
When I searched for similar issues, they told the problem was that the optimizer was not saved (Because the reference github page didn't save the optimizer, so did I. Is it true?
My losses after loading

Output:
if you see this line!
you are Decaying the learning rate of each parameter group by gamma.
This has altered your learning rate as you had reached 100th epoch. and moreover you had not saved your optimizer state while saving your model.
This made your code to start with the starting lr i.e 0.1 after resuming your training.
And this spiked your loss again.
Vote if you find this useful


Instruction: How to learn two functions simultaneously in using python (either pytorch or tensorflow)?

Input:
I have three series of observations, namely Y, T, and X. I would like to study the differences between the predicted values of the two models. The first model is to learn g such that Y=g(T, X). The second model is to learn L and f such that Y=L(T)f(X). I have no problem in learning the first model using the PyTorch package or the Tensorflow package. However, I am not sure how to learn L and f. In using the PyTorch package, I can set up two feedforward MLPs with different hidden layers and inputs. For simplicity, I define a Feedforward MLP class as follows:
class Feedforward(t.nn.Module): # the definition of a feedforward neural network
    # Basic definition
    def __init__(self, input_size, hidden_size):
        super(Feedforward, self).__init__()
        self.input_size = input_size
        self.hidden_size  = hidden_size
        self.fc1 = t.nn.Linear(self.input_size, self.hidden_size)
        self.relu = t.nn.ReLU()
        self.fc2 = t.nn.Linear(self.hidden_size, 1)
        self.sigmoid = t.nn.Sigmoid()
    # Advance definition
    def forward(self, x):
        hidden = self.fc1(x)
        relu = self.relu(hidden)
        output = self.fc2(relu)
        output = self.sigmoid(output)
        return output

Suppose L=Feedforward(2,10) and L=Feedforward(3,9). From my understanding, I can only learn either L or f, but not both simultaneously. Is it possible to learn L and f simultaneously using Y, T, and X?

Output:
I may be missing something, but I think you can :
L = Feedforward(2,10)
f = Feedforward(3,9)
L_opt = Adam(L.parameters(), lr=...)
f_opt = Adam(f.parameters(), lr=...)
for (x,t,y) in dataset:
    L.zero_grad()
    f.zero_grad()
    y_pred = L(t)*f(x)
    loss = (y-y_pred)**2
    loss.backward()
    L_opt.step()
    f_opt.step()

You can also fuse them together in one single model :
class ProductModel(t.nn.Module):
    def __init__(self, L, f):
        self.L = L
        self.f = f
    def forward(self, x,t):
        return self.L(t)*self.f(x)

and then train this model like you trained g


Instruction: dimension extension with pytorch tensors

Input:
What is the way dimension extension with pytorch tensors?
-before:
torch.Size([3, 3, 3])
tensor([[[ 0.,  1.,  2.],
         [ 3.,  4.,  5.],
         [ 6.,  7.,  8.]],

        [[ 9., 10., 11.],
         [12., 13., 14.],
         [15., 16., 17.]],

        [[18., 19., 20.],
         [21., 22., 23.],
         [24., 25., 26.]]], device='cuda:0', dtype=torch.float64)

-after:
torch.Size([2, 3, 3, 3])
tensor([[[[ 0.,  1.,  2.],
          [ 3.,  4.,  5.],
          [ 6.,  7.,  8.]],

         [[ 9., 10., 11.],
          [12., 13., 14.],
          [15., 16., 17.]],

         [[18., 19., 20.],
          [21., 22., 23.],
          [24., 25., 26.]]],

        [[[0., 1., 2.],
          [ 3.,  4.,  5.],
          [ 6.,  7.,  8.]],

         [[ 9., 10., 11.],
          [12., 13., 14.],
          [15., 16., 17.]],

         [[18., 19., 20.],
          [21., 22., 23.],
          [24., 25., 26.]]]], device='cuda:0', dtype=torch.float64)

under numpy would work like this:
b =  np.broadcast_to(a1[None, :,:,:], (2,3,3,3))

How does this work under pytorch? I want to take advantage of the gpu. Thanks in advance for the help!

Output:
A new dimension can be added with unsqeeze (0 used below to specify the first dimension, i.e., position 0), followed by repeating the data twice along that dimension (and once, i.e., no repetitions, along the other dimensions).
before = torch.tensor(..., dtype=torch.float64, device='cuda:0')
after = before.unsqueeze(0).repeat(2, 1, 1, 1)


Instruction: Should the parameters of the BatchNorm layer be regularized

Input: When I use pytorch to train my CNN, the L2 regularization will be used to panalize the parameters in the model. But the pytorch code "weight decay" will use L2 to all the parameters which can be updated. Should the parameters in BatchNorm Layers be panalized by L2, too?

Output:
If you are using BatchNorm right after nn.Conv2d or nn.Linear, you can "fold" the learned weight and bias into the conv/linear layer.
hence, the learned weigh and bias has a direct effect on the actual L2 norm of the "effective" weights of your network.
Therefore, if you are using L2 regularization to push your weights towards zero - you must also regularize the batch norm parameters.


Instruction: How to load a dataset starting from list of images Pytorch

Input:
I have a service that receives images in a binary format from another service (let's call it service B):
from PIL import Image

img_list = []
img_bin = get_image_from_service_B()
image = Image.open(io.BytesIO(img_bin)) # Convert bytes to image using PIL

When an image is successfully converted thanks to PIL it is also appended to a list of images.
img_list.append(image)

When I've enough images I want to load my list of images using Pytorch as if it was a dataset
if img_list.__len__() == 500:
     ### Load dataset and do a transform operation on the data

In a previous version of the software the requirement was simply to retrieve the images from a folder, so it was quite simple to load all the images
my_dataset = datasets.ImageFolder("path/to/images/folder/", transform=transform)
dataset_iterator = DataLoader(my_dataset, batch_size=1)

Now my issue is how to perform the transform and load the dataset from a list.

Output:
You can simply write a custom dataset:
class MyDataset(torch.utils.data.Dataset):
    def __init__(self, img_list, augmentations):
        super(MyDataset, self).__init__()
        self.img_list = img_list
        self.augmentations = augmentations

    def __len__(self):
        return len(self.img_list)

    def __getitem__(self, idx):
        img = self.img_list[idx]
        return self.augmentations(img)

You can now plug this custom dataset into DataLoader and you are done.


Instruction: Accuracy and loss does not change with RMSprop optimizer

Input:
The dataset is CIFAR10. I've created a VGG-like network:
class FirstModel(nn.Module):
    def __init__(self):
        super(FirstModel, self).__init__()
        self.vgg1 = nn.Sequential(
            nn.Conv2d(3, 16, 3, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.Conv2d(16, 16, 3, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(2,2),
            nn.Dropout(0.2)
        )

        self.vgg2 = nn.Sequential(
            nn.Conv2d(16, 32, 3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(32, 32, 3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2,2),
            nn.Dropout(0.2)
        )

        self.vgg3 = nn.Sequential(
            nn.Conv2d(32, 64, 3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(64, 64, 3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2,2),
            nn.Dropout(0.2)
        )

        self.fc1 = nn.Linear(4 * 4 * 64, 4096)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(4096, 4096)
        self.fc3 = nn.Linear(4096, 10)
        self.softmax = nn.Softmax()
        self.dropout = nn.Dropout(0.5)

    def forward(self, x):
        x = self.vgg3(self.vgg2(self.vgg1(x)))
        x = nn.Flatten()(x)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.relu(self.fc2(x))
        x = self.dropout(x)
        x = self.softmax(self.fc3(x))

        return x

Then I train it and visualize loss and accuracy:
import matplotlib.pyplot as plt
from IPython.display import clear_output

def plot_history(train_history, val_history, title='loss'):
    plt.figure()
    plt.title('{}'.format(title))
    plt.plot(train_history, label='train', zorder=1)

    points = np.array(val_history)
    steps = list(range(0, len(train_history) + 1, int(len(train_history) / len(val_history))))[1:]

    plt.scatter(steps, val_history, marker='*', s=180, c='red', label='val', zorder=2)
    plt.xlabel('train steps')

    plt.legend(loc='best')
    plt.grid()

    plt.show()

def train_model(model, optimizer, train_dataloader, test_dataloader):
      criterion = nn.CrossEntropyLoss()

      train_loss_log = []
      train_acc_log = []
      val_loss_log = []
      val_acc_log = []

      for epoch in range(NUM_EPOCH):

        model.train()

        train_loss = 0.
        train_size = 0
        train_acc = 0.

        for inputs, labels in train_dataloader:

          inputs, labels = inputs.to(device), labels.to(device)

          optimizer.zero_grad()
          y_pred = model(inputs)

          loss = criterion(y_pred, labels)
          loss.backward()
          optimizer.step()

          train_loss += loss.item()
          train_size += y_pred.size(0)
          train_loss_log.append(loss.data / y_pred.size(0))

          _, pred_classes = torch.max(y_pred, 1)
          train_acc += (pred_classes == labels).sum().item()
          train_acc_log.append(np.mean((pred_classes == labels).cpu().numpy()))

        # блок validation
        val_loss = 0.
        val_size = 0
        val_acc = 0.

        model.eval()
        with torch.no_grad():
            for inputs, labels in test_dataloader:
                inputs, labels = inputs.to(device), labels.to(device)
                y_pred = model(inputs)
                loss = criterion(y_pred, labels)
                val_loss += loss.item()
                val_size += y_pred.size(0)

                _, pred_classes = torch.max(y_pred, 1)
                val_acc += (pred_classes == labels).sum().item()

        val_loss_log.append(val_loss/val_size)
        val_acc_log.append(val_acc/val_size)

        clear_output()
        plot_history(train_loss_log, val_loss_log, 'loss')
        plot_history(train_acc_log, val_acc_log, 'accuracy')

        print('Train loss:', train_loss / train_size)
        print('Train acc:', train_acc / train_size)
        print('Val loss:', val_loss / val_size)
        print('Val acc:', val_acc / val_size)

Then I train the model:
first_model = FirstModel()
first_model.to(device)

optimizer = optim.RMSprop(first_model.parameters(), lr=0.001, momentum=0.9)

train_model(first_model_rms, optimizer, train_dataloader, test_dataloader)

The loss and accuracy do not change (accuracy at level of 0.1). However, if the optimizer is SGD with momentum everything works fine (loss and accuracy change). I've already tried to change momentum and lr, but it does not help.
What should be fixed? Would be grateful for any possible advice!

Output: So first of all, you don't have to use softmax in the "model" as it is done by the nn.CrossEntropyLoss, and I also think that the RMSprop doesn't work with momentum.


Instruction: Sentence embedding using T5

Input:
I would like to use state-of-the-art LM T5 to get sentence embedding vector.
I found this repository https://github.com/UKPLab/sentence-transformers
As I know, in BERT I should take the first token as [CLS] token, and it will be the sentence embedding.
In this repository I see the same behaviour on T5 model:
cls_tokens = output_tokens[:, 0, :]  # CLS token is first token

Does this behaviour correct? I have taken encoder from T5 and encoded two phrases with it:
"I live in the kindergarden"
"Yes, I live in the kindergarden"

The cosine similarity between them was only "0.2420".
I just need to understand how sentence embedding works - should I train network to find similarity to reach correct results? Or I it is enough of base pretrained language model?

Output:
In order to obtain the sentence embedding from the T5, you need to take the take the last_hidden_state from the T5 encoder output:
model.encoder(input_ids=s, attention_mask=attn, return_dict=True)
pooled_sentence = output.last_hidden_state # shape is [batch_size, seq_len, hidden_size]
# pooled_sentence will represent the embeddings for each word in the sentence
# you need to sum/average the pooled_sentence
pooled_sentence = torch.mean(pooled_sentence, dim=1)

You have now a sentence embeddings from T5


Instruction: Passing 3 dimensional and one dimensional features to neural network with PyTorch Dataloader

Input:
I have examples with size 2x8x8 as tensor and I'm using PyTorch Dataloader for them. But now I want to add an additional 1 dim tensor with size 1 (a single number) as input too.
So I have two input parameters for the neural network, a multidimensional for convolutional layers and an additional one that I will concatenate later.
Probably I could use two dataloader, for every tensor shape one, but than I could not shuffle them.
How can I use a single PyTorch Dataloader for this two different input tensors?

Output:
This is not about the dataloader, this should be done in the your dataset. Implement your own dataset by making it inherit from torch.util.data.Dataset (you need to implement __len__ and __getitem__). Make your __getitem__method return both your tensors, and you should be fine.
You can follow this tutorial if  you need.


Instruction: Problem running GRU model; missing argument for forward()

Input:
I am working on a GRU and when I try to make predictions I get an error indicating that I need to define h for forward(). I have tried several things and ran out of patience after googling and scouring stack overflow for hours.
This is the class:
class GRUNet(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, n_layers, drop_prob = 0.2):
        super(GRUNet, self).__init__()
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers

        self.gru = nn.GRU(input_dim, hidden_dim, n_layers, batch_first=True, dropout=drop_prob)
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.relu = nn.ReLU()

    def forward(self, x, h):
        out, h = self.gru(x,h)
        out = self.fc(self.relu(out[:,-1]))
        return out, h

    def init_hidden(self, batch_size):
        weight = next(self.parameters()).data
        hidden = weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().to(device)
        return hidden

and then this is where I load the model and try to make a prediction. Both of these are in the same script.
inputs = np.load('.//Pred//input_list.npy')
print(inputs.ndim, inputs.shape)
Gmodel = GRUNet(24,256,1,2)
Gmodel = torch.load('.//GRU//GRU_1028_48.pkl')
Gmodel.eval()
pred = Gmodel(inputs)

Without any other arguments to Gmodel I get the following:
Traceback (most recent call last):
  File ".\grunet.py", line 136, in <module>
    pred = Gmodel(inputs)
  File "C:\Users\ryang\Anaconda-3\envs\tf-gpu\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'h'

Output:
You need to provide the hidden state as well which is usually initially all zeros or simply None!
That is you either need to explicitly provide one like this :
hidden_state = torch.zeros(size=(num_layers*direction, batch_size, hidden_dim)).to(device)
pred = Gmodel(inputs, hidden_state)

or simply do :
hidden_state = None
pred = Gmodel(inputs, hidden_state)


Instruction: Adding class objects to Pytorch Dataloader: batch must contain tensors

Input:
I have a custom Pytorch dataset that returns a dictionary containing a class object "queries".
class QueryDataset(torch.utils.data.Dataset):

    def __init__(self, queries, values, targets):
        super(QueryDataset).__init__()
        self.queries = queries
        self.values = values
        self.targets = targets

    def __len__(self):
        return self.values.shape[0]

    def __getitem__(self, idx):
        sample = DeviceDict({'query': self.queries[idx],
                             "values": self.values[idx],
                             "targets": self.targets[idx]})
        return sample

The problem is that when I put the queries in a data loader I get default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'query.Query'>. Is there a way to have a class object in my data loader? It blows up at next(iterator) in the code below.
train_queries = QueryDataset(train_queries)
train_loader = torch.utils.data.DataLoader(train_queries,
                                           batch_size=10],
                                           shuffle=True,
                                           drop_last=False)
for i in range(epochs):
    iterator = iter(train_loader)
    for i in range(len(train_loader)):
        batch = next(iterator)
        out = model(batch)
        loss = criterion(out["pred"], batch["targets"])
        self.optimizer.zero_grad()
        loss.sum().backward()
        self.optimizer.step()

Output:
You need to define your own colate_fn in order to do this.
A sloppy approach just to show you how stuff works here, would be something like this:
import torch
class DeviceDict:
    def __init__(self, data):
        self.data = data

    def print_data(self):
        print(self.data)

class QueryDataset(torch.utils.data.Dataset):

    def __init__(self, queries, values, targets):
        super(QueryDataset).__init__()
        self.queries = queries
        self.values = values
        self.targets = targets

    def __len__(self):
        return 5

    def __getitem__(self, idx):
        sample = {'query': self.queries[idx],
                 "values": self.values[idx],
                 "targets": self.targets[idx]}
        return sample

def custom_collate(dict):
    return DeviceDict(dict)

dt = QueryDataset("q","v","t")
dl = torch.utils.data.DataLoader(dtt,batch_size=1,collate_fn=custom_collate)
t = next(iter(dl))
t.print_data()

Basically colate_fn allows you to achieve custom batching or adding support for custom data types as explained in the link I previously provided.
As you see it just shows the concept, you need to change it based on your own needs.


Instruction: numpy.empty() data type not understood error on Google colab

Input:
numpy.empty() returns data type not understood exception on Google colab's default numpy library. I checked all questions in stackoverflow but didnt see related to this problem since i am using google colab. Here is full code and exception output:
Full code:
def diffMask(img1=None, img2=None, opt=None, dataset=None, args=None):
    netG = args[0]
    netB = args[1]
    netD = args[2]
    f = args[3]
    res_path = opt.results_Stage3
    res_folders = ['temp_masks',
                   'temp_Stage2',
                   'temp_ref',
                   'temp_diff',
                   'temp_Stage3',
                   'temp_skel',
                   'temp_res',
                   'temp_Stage1',
                   'temp_src']
    for x in res_folders:
        if os.path.isdir("{}{}".format(res_path, x)) == False:
            os.mkdir("{}{}".format(res_path, x))
    save_masks = "{}{}".format(res_path, "temp_masks")
    save_Stage2 = "{}{}".format(res_path, "temp_Stage2")
    save_ref = "{}{}".format(res_path, "temp_ref")
    save_diff = "{}{}".format(res_path, "temp_diff")
    save_Stage3 = "{}{}".format(res_path, "temp_Stage3")
    save_skel = "{}{}".format(res_path, "temp_skel")
    save_res = "{}{}".format(res_path, "temp_res")
    save_Stage1 = "{}{}".format(res_path, "temp_Stage1")
    save_src = "{}{}".format(res_path, "temp_src")

    resize2 = transforms.Resize(size=(128, 128))
    src, mask, style_img, target, gt_cloth, skel, cloth = dataset.get_img("{}_0.jpg".format(img1[:-6]),
                                                                          "{}_1.jpg".format(img1[:-6]))
    src, mask, style_img, target, gt_cloth, skel, cloth = src.unsqueeze(0), mask.unsqueeze(0), style_img.unsqueeze(
        0), target.unsqueeze(0), gt_cloth.unsqueeze(0), skel.unsqueeze(0), cloth.unsqueeze(0)  # , face.unsqueeze(0)
    src1, mask1, style_img1, target1, gt_cloth1, skel1, cloth1 = Variable(src.cuda()), Variable(mask.cuda()), Variable(
        style_img.cuda()), Variable(target.cuda()), Variable(gt_cloth.cuda()), Variable(skel.cuda()), Variable(
        cloth.cuda())  # , Variable(face.cuda())
    src, mask, style_img, target, gt_cloth, skel, cloth = dataset.get_img("{}_0.jpg".format(img2[:-6]),
                                                                          "{}_1.jpg".format(img2[:-6]))
    src, mask, style_img, target, gt_cloth, skel, cloth = src.unsqueeze(0), mask.unsqueeze(0), style_img.unsqueeze(
        0), target.unsqueeze(0), gt_cloth.unsqueeze(0), skel.unsqueeze(0), cloth.unsqueeze(0)  # , face.unsqueeze(0)
    src2, mask2, style_img2, target2, gt_cloth2, skel2, cloth2 = Variable(src.cuda()), Variable(mask.cuda()), Variable(
        style_img.cuda()), Variable(target.cuda()), Variable(gt_cloth.cuda()), Variable(skel.cuda()), Variable(
        cloth.cuda())

    gen_targ_Stage1, s_128, s_64, s_32, s_16, s_8, s_4 = netG(skel1, cloth2)  # gen_targ11 is structural change cloth
    gen_targ_Stage2, s_128, s_64, s_32, s_16, s_8, s_4 = netB(src1, gen_targ_Stage1,
                                                              skel1)  # gen_targ12 is Stage2 image

    # saving structural
    pic_Stage2 = (torch.cat([gen_targ_Stage2], dim=0).data + 1) / 2.0
    #     save_dir = "/home/np9207/PolyGan_res/temp_Stage2/"
    save_image(pic_Stage2, '%s/%d_%s_%d.jpg' % (save_Stage2, f, img1[:-6], 0), nrow=1)

    msk1 = mask1[0, :, :, :].detach().cpu().permute(1, 2, 0)
    plt.imsave("{}/{}_{}_mask.jpg".format(save_masks, f, img1[:-6]), msk1, cmap="gray")

I got exception on this line:
plt.imsave("{}/{}_{}_mask.jpg".format(save_masks, f, img1[:-6]), msk1, cmap="gray")

Exception Output:
Traceback (most recent call last):
  File "test.py", line 208, in test
    diffMask(image1, image2, opt, test_loader, args)
  File "test.py", line 96, in diffMask
    plt.imsave("{}/{}_{}_mask.jpg".format(save_masks, f, img1[:-6]), msk1, cmap="gray")
  File "/usr/local/lib/python3.6/dist-packages/matplotlib/pyplot.py", line 2066, in imsave
    return matplotlib.image.imsave(fname, arr, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/matplotlib/image.py", line 1550, in imsave
    rgba = sm.to_rgba(arr, bytes=True)
  File "/usr/local/lib/python3.6/dist-packages/matplotlib/cm.py", line 217, in to_rgba
    xx = np.empty(shape=(m, n, 4), dtype=x.dtype)
TypeError: data type not understood

Numpy Version : 1.18.5
Thank you in advance.

Output:
Thanks to @user2357112 @Rika and @hpaulj 's great helps, I found the cause of the problem and here is solution for those who is struggling  the same issue:
msk1 = msk1.numpy()

But then (probably) you will get ValueError: Floating point image RGB values must be in the 0..1 range. exception. You have to normalize your array. Thanks to him
def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalized data
    """
    return np.array((x - np.min(x)) / (np.max(x) - np.min(x)))

msk1 = normalize(msk1.numpy())


Instruction: Activate pytorch network nodes without variable assignments?

Input:
In tutorials for PyTorch networks, we typically see an implementation such as:
from torch.nn.functional import hardtanh, sigmoid
import torch.nn as nn

class great_network(nn.Module):
    def __init__(self):
        super(great_network, self).__init__()
        self.layer1 = nn.Conv2d(2, 2, 3)
        self.pool_1 = nn.MaxPool2d(1, 1)
        self.layer3 = nn.ConvTranspose2d(2, 2, 3)
        self.out_layer = nn.Conv2d(1, 1, 3)

    def forward(self, x):
        x = hardtanh(self.layer1(x))
        x = self.pool_1(x)
        x = hardtanh(self.layer3(x))
        x = sigmoid(self.out_layer(x))
        return x

net = great_network()
print(net)

great_network(
  (layer1): Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
  (pool_1): MaxPool2d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)
  (layer3): ConvTranspose2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
  (out_layer): Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1))
)

If I were to dynamically change the size of this network to run multiple experiments, I'd have to emulate the above code (resembling a data clump code bloat) without multiple assignments.
Something like this might do:
from torch.nn.functional import hardtanh, sigmoid
import torch.nn as nn
import numpy as np

class not_so_great_network(nn.Module):
    def __init__(self, n):
        super(not_so_great_network, self).__init__()
        self.pre_layers = self.generate_pre_layers(n)
        self.post_layers = self.generate_post_layers(n)
        self.pool = nn.MaxPool2d(1, 1)
        self.out = nn.Conv2d(1, 1, 3)

    def generate_pre_layers(self, layer_num):
        layers = np.empty(layer_num, dtype = object)
        for lay in range(0, len(layers)):
            layers[lay] = nn.Conv2d(2, 2, 3)
        return layers

    def generate_post_layers(self, layer_num):
        layers = np.empty(layer_num, dtype = object)
        for lay in range(0, len(layers)):
            layers[lay] = nn.Conv2d(2, 2, 3)
        return layers

    def forward(self, x):
        for pre in self.pre_layers:
            x = hardtanh(pre(x))
            x = self.pool(x)

        for post in self.post_layers:
            x = hardtanh(post(x))

        x = sigmoid(self.out(x))

        return x

However, not all layers are there:
if __name__ == '__main__':
    layer_num = 5
    net = not_so_great_network(layer_num)
    print(net)

not_so_great_network(
  (pool): MaxPool2d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)
  (out): Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1))
)

I am not assigning the variables because this could be more powerful if I can generate networks of varying sizes without copying and pasting. How can I emulate the output so that I may activate nodes with activation functions later?

Output:
An alternative would be using ModuleList:
from torch import nn
from torch.nn.functional import hardtanh, sigmoid

class maybe_great_network(nn.Module):
    def __init__(self, n):
        super().__init__()
        self.pre_layers = self.generate_pre_layers(n)
        self.post_layers = self.generate_post_layers(n)
        self.pool = nn.MaxPool2d(1, 1)
        self.out = nn.Conv2d(1, 1, 3)

    def generate_pre_layers(self, layer_num):
        return nn.ModuleList([
            nn.Conv2d(2, 2, 3)
            for l in range(0, layer_num)
        ])

    def generate_post_layers(self, layer_num):
        return nn.ModuleList([
            nn.Conv2d(2, 2, 3)
            for l in range(0, layer_num)
        ])

    def forward(self, x):
        for pre in self.pre_layers:
            x = hardtanh(pre(x))
            x = self.pool(x)
        for post in self.post_layers:
            x = hardtanh(post(x))
        x = sigmoid(self.out(x))
        return x

Then:
>>> m = maybe_great_network(3)
>>> m
maybe_great_network(
  (pre_layers): ModuleList(
    (0): Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
    (1): Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
    (2): Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
  )
  (post_layers): ModuleList(
    (0): Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
    (1): Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
    (2): Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
  )
  (pool): MaxPool2d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)
  (out): Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1))
)


Instruction: Filter out NaN values from a PyTorch N-Dimensional tensor

Input:
This question is very similar to filtering np.nan values from pytorch in a -Dimensional tensor. The difference is that I want to apply the same concept to tensors of 2 or higher dimensions.
I have a tensor that looks like this:
import torch

tensor = torch.Tensor(
[[1, 1, 1, 1, 1],
 [float('nan'), float('nan'), float('nan'), float('nan'), float('nan')],
 [2, 2, 2, 2, 2]]
)

>>> tensor.shape
>>> [3, 5]

I would like to find the most pythonic / PyTorch way of to filter out (remove) the rows of the tensor which are nan. By filtering this tensor along the first (0th axis) I want to obtain a filtered_tensor which looks like this:
>>> print(filtered_tensor)
>>> torch.Tensor(
[[1, 1, 1, 1, 1],
 [2, 2, 2, 2, 2]]
)
>>> filtered_tensor.shape
>>> [2, 5]

Output:
Use PyTorch's isnan() together with any() to slice tensor's rows using the obtained boolean mask as follows:
filtered_tensor = tensor[~torch.any(tensor.isnan(),dim=1)]

Note that this will drop any row that has a nan value in it. If you want to drop only rows where all values are nan replace torch.any with torch.all.
For an N-dimensional tensor you could just flatten all the dims apart from the first dim and apply the same procedure as above:
#Flatten:
shape = tensor.shape
tensor_reshaped = tensor.reshape(shape[0],-1)
#Drop all rows containing any nan:
tensor_reshaped = tensor_reshaped[~torch.any(tensor_reshaped.isnan(),dim=1)]
#Reshape back:
tensor = tensor_reshaped.reshape(tensor_reshaped.shape[0],*shape[1:])


Instruction: ‘DataParallel’ object has no attribute ‘conv1’

Input:
I am trying to visualize cnn network features map for conv1 layer based on the code and architecture below. It’s working properly without DataParallel, but when I am activating model = nn.DataParallel(model) it raised with error: ‘DataParallel’ object has no attribute ‘conv1’. Any suggestion appreciated.
class Model(nn.Module):
    def __init__(self, kernel, num_filters, res = ResidualBlock):
        super(Model, self).__init__()

        self.conv0 = nn.Sequential(
            nn.Conv2d(4, num_filters, kernel_size = kernel*3,
                       padding = 4),
            nn.BatchNorm2d(num_filters),
            nn.ReLU(inplace=True))

        self.conv1 = nn.Sequential(
            nn.Conv2d(num_filters, num_filters*2, kernel_size = kernel,
                      stride=2, padding = 1),
            nn.BatchNorm2d(num_filters*2),
            nn.ReLU(inplace=True))

        self.conv2 = nn.Sequential(
            nn.Conv2d(num_filters*2, num_filters*4, kernel_size = kernel, stride=2, padding = 1),
            nn.BatchNorm2d(num_filters*4),
            nn.ReLU(inplace=True))

        self.tsconv0 = nn.Sequential(
            nn.ConvTranspose2d(num_filters*4, num_filters*2, kernel_size = kernel, padding = 1),
            nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(num_filters*2))

        self.tsconv1 = nn.Sequential(
            nn.ConvTranspose2d(num_filters*2, num_filters, kernel_size = kernel, padding = 1),
            nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(num_filters))

        self.tsconv2 = nn.Sequential(
            nn.Conv2d(num_filters, 1, kernel_size = kernel*3, padding = 4, bias=False),
            nn.ReLU(inplace=True))

model = Model(kernel, num_filters)
model = nn.DataParallel(model)

The code for feature map visualization:
def get_activation(name):
    def hook(model, x_train_batch, y_train_pred):
        activation[name] = y_train_pred.detach()
    return hook

model.conv3.register_forward_hook(get_activation('conv3'))
x_train_batch[0,0,:,:]
y_train_pred = model(x_train_batch)

act = activation['conv3'].squeeze()
act1 = act.cpu().detach().numpy()
act=act[0,:,:,:]

fig, axarr = plt.subplots(6,16)
k = 0
for idx in range(act.size(0)//16):
    for idy in range(act.size(0)//6):
        axarr[idx, idy].imshow(act[k])
        k += 1

Output: When you use DataParallel, add an extra module there. instead of doing model.conv3. simply write model.module.conv3.


Instruction: Pytorch input tensor size with wrong dimension Conv1D

Input:
 def train(epoch):
      model.train()
      train_loss = 0

  for batch_idx, (data, _) in enumerate(train_loader):
    data = data[None, :, :]
    print(data.size())    # something seems to change between here

    data = data.to(device)
    optimizer.zero_grad()
    recon_batch, mu, logvar = model(data) # and here???

    loss = loss_function(recon_batch, data, mu, logvar)
    loss.backward()
    train_loss += loss.item()

    optimizer.step()

    if batch_idx % 1000 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader),
                loss.item() / len(data)))

  print('====> Epoch: {} Average loss: {:.4f}'.format(epoch, train_loss / len(train_loader.dataset)))

for epoch in range(1, 4):
        train(epoch)

This is very strange looking at the training loop it does recognize that the size is [1,1,1998] but then something changes after it is sent to the device?
    torch.Size([1, 1, 1998])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-138-70cca679f91a> in <module>()
     27
     28 for epoch in range(1, 4):
---> 29         train(epoch)

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    255                             _single(0), self.dilation, self.groups)
    256         return F.conv1d(input, self.weight, self.bias, self.stride,
--> 257                         self.padding, self.dilation, self.groups)
    258
    259

RuntimeError: Expected 3-dimensional input for 3-dimensional weight [12, 1, 1], but got 2-dimensional input of size [1, 1998] instead

Also here is my model (I recognize there is likely a couple of other issues here but I am asking about the tensor size not registering)
class VAE(nn.Module):
  def __init__(self):
    super(VAE, self).__init__()

    self.conv1 = nn.Conv1d( 1,12, kernel_size=1,stride=5,padding=0)
    self.conv1_drop = nn.Dropout2d()
    self.pool1 = nn.MaxPool1d(kernel_size=3, stride=2)

    self.fc21 = nn.Linear(198, 1)
    self.fc22 = nn.Linear(198, 1)

    self.fc3 = nn.Linear(1, 198)
    self.fc4 = nn.Linear(198, 1998)

  def encode(self, x):
    h1 = self.conv1(x)
    h1 = self.conv1_drop(h1)
    h1 = self.pool1(h1)
    h1 = F.relu(h1)
    h1 = h1.view(1, -1) # 1 is the batch size
    return self.fc21(h1), self.fc22(h1)

  def reparameterize(self, mu, logvar):
    std = torch.exp(0.5*logvar)
    eps = torch.rand_like(std)
    return mu + eps*std

  def decode(self, z):
    h3 = F.relu(self.fc3(z))
    return torch.sigmoid(self.fc4(h3))

  def forward(self, x):
    mu, logvar = self.encode(x.view(-1, 1998))
    z = self.reparameterize(mu, logvar)
    return self.decode(z), mu, logvar

So why doesn't Pytorch keep the dimensions after reshaping and would that be the correct tensor size if it did?

Output: I just found my mistake when I call forward() I am doing self.encode(x.view(-1,1998)) which is reshaping the tensor.


Instruction: Loading data from Custom Data-Loader in pytorch only if the data specifies a certain condition

Input:
I have a CSV file with filename in the first column and a label for the filename in the second column. I also have a third column, which specifies something about the data (whether the data meets a specific condition). It will look something like,
+-----------------------------+
| Filepath 1   Label 1    'n' |
|                             |
+-----------------------------+
| Filepath 2   Label 2    'n' |
|                             |
|                             |
+-----------------------------+
| Filepath 3   Label 3     'n'|
|                             |
+-----------------------------+
| Filepath 4   Label 4     'y'|
+------------------------------+

I want to be able to load the custom dataset using getitem only when attribute column == 'y'. However, I get the following error:
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>
My code is as follows:
'''
class InterDataset(Dataset):
  def __init__(self, csv_file, mode, root_dir = None, transform = None, run = None):
    self.annotations = pd.read_csv(csv_file, header = None)
    self.root_dir = root_dir
    self.transform = transform
    self.mode = mode
    self.run = run

  def __len__(self):
    return len(self.annotations)

  def __getitem__(self, index):
    if self.mode == 'train':
        if (self.annotations.iloc[index, 2] == 'n'):
                    img_path = self.annotations.iloc[index,0]
                    image = cv2.imread(img_path,1)

                    y_label = self.annotations.iloc[index,1]

                    if self.transform:
                        image = self.transform(image)
                    if (index+1)%300 == 0:
                        print('Loop {0} done'.format(index))
                    return [image, y_label]

'''

Output:
You get that error because the dataloader has to return something. Here are three solutions:

There is a libary called nonechucks which lets you create dataloaders in which you can skip samples.
Usually you could preprocess/clean your data and kick the unwanted samples out.
You could return some indicator that the sample is unwanted, for example

if "y":
    return data, target
else:
    return -1

And then you could check in your train loop if the "data" is -1 and skip the iteration.
I hope this was helpful :)


Instruction: Unable to Load Images to train model on Custom Datasets

Input:
I have just stuck with Image Instance Segmentation for a while. I am trying to train the Yolact model for my custom data. Here is some brief information about what I have done so far

I have annotated the image using labelme annotation tool
I have converted annotation file for each (train & validation data) using labelme2coco -> train.json & test.json
I made changes in the cofig.py file as needed and expected by yolact
As I was following this repository I encountered an error Argument 'bb' has incorrect type to which I have solved with the approach stated in this closed issue

After completing the above task I am stuck here with below-stated issue.
Scaling parameters by 0.12 to account for a batch size of 1.
Per-GPU batch size is less than the recommended limit for batch norm. Disabling batch norm.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py:165: UserWarning: 'lat_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py:165: UserWarning: 'pred_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py:165: UserWarning: 'downsample_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
Initializing weights...
Begin training!

**_Traceback (most recent call last):_**
  File "train.py", line 504, in <module>
    train()
  File "train.py", line 270, in train
    for datum in data_loader:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 989, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1014, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
**AssertionError: Caught AssertionError in DataLoader worker process 0.**
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/content/yolact/data/coco.py", line 94, in __getitem__
    im, gt, masks, h, w, num_crowds = self.pull_item(index)
  File "/content/yolact/data/coco.py", line 141, in pull_item
    assert osp.exists(path), 'Image path does not exist: {}'.format(path)
**AssertionError: Image path does not exist: data/YolaDataset/train/6.JPG**

NOTE:I have moved my test & train data to yolact/data/YolactDataset ( also CWD is /yolact/ )
Here is the Log file
Yolact Leather Defect Config.log
Here is the content of config.py
yolact_leather_defect_dataset = Config({
    'name': 'Yolact Leather Defect',

    # Training images and annotations
    'train_images': './data/YolaDataset/train/',
    'train_info':   './data/train.json',

    # Validation images and annotations.
    'valid_images': './data/YolaDataset/test/',
    'valid_info':   './data/test.json',

    # Whether or not to load GT. If this is False, eval.py quantitative evaluation won't work.
    'has_gt': True,

    # A list of names for each of you classes.
    'class_names': ("FC", "LF", "OC", "PM"),

    # COCO class ids aren't sequential, so this is a bandage fix. If your ids aren't sequential,
    # provide a map from category_id -> index in class_names + 1 (the +1 is there because it's 1-indexed).
    # If not specified, this just assumes category ids start at 1 and increase sequentially.
    'label_map': {
        1:1, 2:2, 3:3, 4:4,
    }
})

yolact_leather_defect_config = coco_base_config.copy({
    'name': 'Yolact Leather Defect Config',

    # Dataset stuff
    'dataset': yolact_leather_defect_dataset,
    'num_classes': len(yolact_leather_defect_dataset.class_names) + 1,

    # Image Size
    'max_size': 550,

    # Training params
    'lr_steps': (280000, 600000, 700000, 750000),
    'max_iter': 800000,

    # Backbone Settings
    'backbone': resnet101_backbone.copy({
        'selected_layers': list(range(1, 4)),
        'use_pixel_scales': True,
        'preapply_sqrt': False,
        'use_square_anchors': True, # This is for backward compatability with a bug

        'pred_aspect_ratios': [ [[1, 1/2, 2]] ]*5,
        'pred_scales': [[24], [48], [96], [192], [384]],
    }),

    # FPN Settings
    'fpn': fpn_base.copy({
        'use_conv_downsample': True,
        'num_downsample': 2,
    }),

    # Mask Settings
    'mask_type': mask_type.lincomb,
    'mask_alpha': 6.125,
    'mask_proto_src': 0,
    'mask_proto_net': [(256, 3, {'padding': 1})] * 3 + [(None, -2, {}), (256, 3, {'padding': 1})] + [(32, 1, {})],
    'mask_proto_normalize_emulate_roi_pooling': True,

    # Other stuff
    'share_prediction_module': True,
    'extra_head_net': [(256, 3, {'padding': 1})],

    'positive_iou_threshold': 0.5,
    'negative_iou_threshold': 0.4,

    'crowd_iou_threshold': 0.7,

    'use_semantic_segmentation_loss': True,
})

File Structure
https://drive.google.com/file/d/1GDaDNBayMsADnxKmbIn5OL9eU17SI6eV/view?usp=sharing
I have tried to resolve the issue to the best can help myself. Any help would be appreciated.
Thank You!
[Originally this issue I have posted on yolact repo.]

Output:
You mispelled your folder names :) YolaDataset needs to be renamed to YolactDataset
# Training images and annotations
'train_images': './data/YolaDataset/train/',
'train_info':   './data/train.json',

# Validation images and annotations.
'valid_images': './data/YolaDataset/test/',
'valid_info':   './data/test.json',


Instruction: BERT-based NER model giving inconsistent prediction when deserialized

Input:
I am trying to train an NER model using the HuggingFace transformers library on Colab cloud GPUs, pickle it and load the model on my own CPU to make predictions.
Code
The model is the following:
from transformers import BertForTokenClassification

model = BertForTokenClassification.from_pretrained(
    "bert-base-cased",
    num_labels=NUM_LABELS,
    output_attentions = False,
    output_hidden_states = False
)

I am using this snippet to save the model on Colab
import torch

torch.save(model.state_dict(), FILENAME)

Then load it on my local CPU using
# Initiating an instance of the model type

model_reload = BertForTokenClassification.from_pretrained(
    "bert-base-cased",
    num_labels=len(tag2idx),
    output_attentions = False,
    output_hidden_states = False
)

# Loading the model
model_reload.load_state_dict(torch.load(FILENAME, map_location='cpu'))
model_reload.eval()

The code snippet used to tokenize the text and make actual predictions is the same both on the Colab GPU notebook instance and my CPU notebook instance.
Expected Behavior
The GPU-trained model behaves correctly and classifies the following tokens perfectly:
O       [CLS]
O       Good
O       morning
O       ,
O       my
O       name
O       is
B-per   John
I-per   Kennedy
O       and
O       I
O       am
O       working
O       at
B-org   Apple
O       in
O       the
O       headquarters
O       of
B-geo   Cupertino
O       [SEP]

Actual Behavior
When loading the model and use it to make predictions on my CPU, the predictions are totally wrong:
I-eve   [CLS]
I-eve   Good
I-eve   morning
I-eve   ,
I-eve   my
I-eve   name
I-eve   is
I-geo   John
B-eve   Kennedy
I-eve   and
I-eve   I
I-eve   am
I-eve   working
I-eve   at
I-gpe   Apple
I-eve   in
I-eve   the
I-eve   headquarters
I-eve   of
B-org   Cupertino
I-eve   [SEP]

Does anyone have ideas why it doesn't work? Did I miss something?

Output:
I fixed it, there were two problems:

The index-label mapping for tokens was wrong, for some reason the list() function worked differently on Colab GPU than my CPU (??)

The snippet used to save the model was not correct, for models based on the huggingface-transformers library you can't use model.save_dict() and load it later, you need to use the save_pretrained() method of your model class, and load it later using from_pretrained().


Instruction: Torch tensor with i-th element as product of all previous

Input:
I have a torch tensor like this one:
A = torch.tensor([1,2,3,4,5,6])

there's in torch a simple way to generate a tensor like this?
[1, 1*2, 1*2*3, 1*2*3*4, 1*2*3*4*5, 1*2*3*4*5*6]

Thanks a lot

Output:
You can use torch.cumprod:
import torch

t = torch.tensor([1, 2, 3, 4, 5, 6])
torch.cumprod(t, dim=0)

which outputs:
tensor([1, 2, 6, 24, 120, 720])


Instruction: multiply many matrices and many vectors pytorch

Input:
I am trying to multiply the following:
A batch of matrices N x M x D
A batch of vectors  N x D x 1
To get a result: N x M x 1
as if I were doing N dot products on M x D D x 1.
I cant seem to find the correct function in PyTorch.
torch.bmm as far as I can tell only works for a batch of vectors and a single matrix. If I have to use torch.einsum then so be it but id rather not!

Output:
It's pretty straightforward and intuitive with einsum:
torch.einsum('ijk, ikl->ijl', mats, vecs)

But your operation is just:
mats @ vecs


Instruction: Pytorch nn.CrossEntropyLoss giving, ValueError: Expected target size (x, y), got torch.Size([x, z]) for 3d tensor

Input:
I am following the example here, where the documentation says:

Input: (N, C) where C = number of classes
Target: (N) where each value is 0 ≤ targets[i] ≤ C−1

And this is the case with the example given for a 2d tensor
loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5)
output = loss(input, target)
output.backward()

But for a 2d tensor, I am getting an error
import torch.nn as nn
import torch
loss = nn.CrossEntropyLoss(ignore_index=0)

inputs = torch.rand(32, 128, 3)
targets = torch.ones(32, 128)

loss(inputs, targets.long())

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-26-61e7f03039a6> in <module>
      7 targets = torch.ones(32, 128)
      8
----> 9 loss(inputs, targets.long())

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    959
    960     def forward(self, input: Tensor, target: Tensor) -> Tensor:
--> 961         return F.cross_entropy(input, target, weight=self.weight,
    962                                ignore_index=self.ignore_index, reduction=self.reduction)
    963

/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2466     if size_average is not None or reduce is not None:
   2467         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2468     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2469
   2470

/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2271         out_size = (n,) + input.size()[2:]
   2272         if target.size()[1:] != input.size()[2:]:
-> 2273             raise ValueError('Expected target size {}, got {}'.format(
   2274                 out_size, target.size()))
   2275         input = input.contiguous()

ValueError: Expected target size (32, 3), got torch.Size([32, 128])

As far as I can tell, I am doing everything right regarding setting up the dimensions. The error message seems to think that I am giving a 2d vector, but I gave it a 3d vector, the 128 size dimension is missing.
Is there something that I didn't set up correctly for this loss function?

Output:
This is what the documentation says about K-dimensional loss:

Can also be used for higher dimension inputs, such as 2D images, by providing an input of size (minibatch, C, d_1, d_2, ..., d_K) with K ≥ 1 , where K is the number of dimensions, and a target of appropriate shape (see below).

The correct input should have a (32, 3, 128) shape, if you have 3 classes:
import torch.nn as nn
import torch
loss = nn.CrossEntropyLoss(ignore_index=0)

inputs = torch.rand(32, 3, 128)
targets = torch.ones(32, 128)

loss(inputs, targets.long())

Or the target should have a (32, 3) shape, if you have 128 classes:
inputs = torch.rand(32, 128, 3)
targets = torch.ones(32, 3)


Instruction: not able to install easyocr (pytorch error)

Input:
I am trying to install easyocr for python using pip
pip install easyocr

but it's not installing
its givig me this in the terminal
ERROR: torchvision 0.5.0 has requirement torch==1.4.0, but you'll have torch 0.1.2.post2 which is incompatible.
Installing collected packages: torch, torchvision, python-bidi, scipy, cycler, kiwisolver, python-dateutil, certifi, pyparsing, matplotlib, PyWavelets, imageio, decorator, networkx, tifffile, scikit-image, easyocr
  Running setup.py install for torch ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\murari\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Murari\\AppData\\Local\\Temp\\pip-install-jl3470hv\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\Murari\\AppData\\Local\\Temp\\pip-install-jl3470hv\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Murari\AppData\Local\Temp\pip-record-8j183bzh\install-record.txt' --single-version-externally-managed --compile
         cwd: C:\Users\Murari\AppData\Local\Temp\pip-install-jl3470hv\torch\
    Complete output (23 lines):
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Murari\AppData\Local\Temp\pip-install-jl3470hv\torch\setup.py", line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "c:\users\murari\appdata\local\programs\python\python37\lib\site-packages\setuptools\__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\Murari\AppData\Local\Temp\pip-install-jl3470hv\torch\setup.py", line 99, in run
        self.run_command('build_deps')
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\Murari\AppData\Local\Temp\pip-install-jl3470hv\torch\setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\murari\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Murari\\AppData\\Local\\Temp\\pip-install-jl3470hv\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\Murari\\AppData\\Local\\Temp\\pip-install-jl3470hv\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Murari\AppData\Local\Temp\pip-record-8j183bzh\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output.
WARNING: You are using pip version 19.2.3, however version 20.2.4 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Output:
I had the same issue and finally fixed it by creating the environment.
Here is the code.
!conda create -n easyocr python=3.8

!conda activate easyocr

pip install easyocr

If the issue got resolved, please vote for the answer.


Instruction: KeyError: 'true ' error when i try to convert labels to 0 and 1

Input:
I am using Arabert (pre trained Bert for Arabic language) for binary classification labeled as true and false, i am trying to change the labels from "true" and "false" to 0 and one i used the code:
import pandas as pd

Data=pd.read_csv("/content/500-instances.csv")
DATA_COLUMN = 'sent'
LABEL_COLUMN = 'label'
Data.columns = [DATA_COLUMN, LABEL_COLUMN]
label_map = {
    'fake' : 0,
    'true' : 1
}

Data[DATA_COLUMN] = Data[DATA_COLUMN].apply(lambda x: preprocess(x, do_farasa_tokenization=False, use_farasapy = False))
Data[LABEL_COLUMN] = Data[LABEL_COLUMN].apply(lambda x: label_map[x])

i get an error: KeyError: 'true ' in the last line.
Do you have any solution?!
Thanks in advance

Output:
These is a trailing space in 'true ', that's why there is no match in label_map, try:
Data[LABEL_COLUMN] = Data[LABEL_COLUMN].apply(lambda x: label_map[x.strip()])

EDIT
If you are not sure what lies in Data[LABEL_COLUMN], I would suggest catching unknown values with a default output value using label_map.get(x.strip(), <DEFAULT_VALUE>).


Instruction: AdamW and Adam with weight decay

Input:
Is there any difference between torch.optim.Adam(weight_decay=0.01) and torch.optim.AdamW(weight_decay=0.01)?
Link to the docs: torch.optim

Output:
Yes, Adam and AdamW weight decay are different.

Hutter pointed out in their paper (Decoupled Weight Decay Regularization) that the way weight decay is implemented in Adam in every library seems to be wrong, and proposed a simple way (which they call AdamW) to fix it.

In Adam, the weight decay is usually implemented by adding wd*w (wd is weight decay here) to the gradients (Ist case), rather than actually subtracting from weights (IInd case).
# Ist: Adam weight decay implementation (L2 regularization)
final_loss = loss + wd * all_weights.pow(2).sum() / 2
# IInd: equivalent to this in SGD
w = w - lr * w.grad - lr * wd * w

These methods are same for vanilla SGD, but as soon as we add momentum, or use a more sophisticated optimizer like Adam, L2 regularization (first equation) and weight decay (second equation) become different.

AdamW follows the second equation for weight decay.
In Adam

weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

In AdamW

weight_decay (float, optional) – weight decay coefficient (default: 1e-2)

Read more on the fastai blog.


Instruction: Why doesn't converting the dtype of a Tensor fix "RuntimeError: expected scalar type Double but found Float"?

Input:
The important part of my code looks like this:
def forward(self, x):
        x = T.tensor(x).to(self.device)
        x = x.type(T.DoubleTensor)
        x = self.conv1(x)
        ...

Yet I still get the error expected scalar type Double but found Float on the last line of that snippet. The line x = x.type(T.DoubleTensor) should fix that, right? I've also tried x = x.double() and  x = T.tensor(x, dtype = T.double).to(self.device) for the earlier line, and I still get the error. I'm at a loss, what am I doing wrong?

Output:
PyTorch expects the input to a layer to have the same device and data type (dtype) as the parameters of the layer. For most layers, including conv layers, the default data type is torch.float32, i.e. a FloatTensor.
To fix your issue you can cast x to be the same type as the weight or bias parameters of the self.conv1 layer (assuming this is a nn.Conv*d layer).
def forward(self, x):
    x = T.tensor(x, device=self.device, dtype=self.conv1.weight.dtype)
    x = self.conv1(x)
    ...

Most likely self.conv1.weight.dtype will just be torch.float32. Unless you've explicitly changed your model parameter types using something like model.to(dtype=torch.float64) then you could equivalently just use
def forward(self, x):
    x = T.tensor(x, device=self.device, dtype=torch.float32)
    x = self.conv1(x)
    ...


Instruction: How to know input/output layer names and sizes for Pytorch model?

Input:
I have Pytorch model.pth using Detectron2's COCO Object Detection Baselines pretrained model R50-FPN.
I am trying to convert the .pth model to onnx.
My code is as follows.
import io
import numpy as np

from torch import nn
import torch.utils.model_zoo as model_zoo
import torch.onnx
from torchvision import models

model = torch.load('output_object_detection/model_final.pth')
x = torch.randn(1, 3, 1080, 1920, requires_grad=True)#0, in_cha, in_h, in_w
torch_out = torch_model(x)
print(model)
torch.onnx.export(torch_model,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "super_resolution.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=10,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['cls_score','bbox_pred'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable lenght axes
                                'output' : {0 : 'batch_size'}})

Is it correct way to convert ONNX model?
If it is the right way, how to know input_names and output_names?
Used netron to see input and output, but the graph doesn't show input/output layers.

Output:
Try this:
import io
import numpy as np
from torch import nn
import torch.utils.model_zoo as model_zoo
import torch.onnx
from torchvision import models

model = torch.load('model_final.pth')
model.eval()
print('Finished loading model!')
print(model)
device = torch.device("cpu" if args.cpu else "cuda")
model = model.to(device)

# ------------------------ export -----------------------------
output_onnx = 'super_resolution.onnx'
print("==> Exporting model to ONNX format at '{}'".format(output_onnx))
input_names = ["input0"]
output_names = ["output0","output1"]
inputs = torch.randn(1, 3, 1080, 1920).to(device)

torch_out = torch.onnx._export(model, inputs, output_onnx, export_params=True, verbose=False,
                               input_names=input_names, output_names=output_names)


Instruction: minimum the cosine similarity of two tensors and output one scalar. Pytorch

Input:
I use Pytorch cosine similarity function as follows. I have two feature vectors and my goal is to make them dissimilar to each other. So, I thought I could minimum their cosine similarity. I have some doubts about the way I have coded. I appreciate your suggestions about the following questions.

I don't know why here are some negative values in val1?

I have done three steps to convert val1 to a scalar. Am I doing it in the right way? Is there any other way?

To minimum the similarity, I have used 1/var1. Is it a standard way to do this? Is it correct if I use 1-var1?
def loss_func(feat1, feat2):
     cosine_loss = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
     val1 = cosine_loss(feat1, feat2).tolist()
     # 1. calculate the absolute values of each element,
     # 2. sum all values together,
     # 3. divide it by the number of values
     val1 = 1/(sum(list(map(abs, val1)))/int(len(val1)))
     val1 = torch.tensor(val1, device='cuda', requires_grad=True)
     return val1

Output:
Do not convert your loss function to a list. This breaks autograd so you won't be able to optimize your model parameters using pytorch.
A loss function is already something to be minimized. If you want to minimize the similarity then you probably just want to return the average cosine similarity. If instead you want minimize the magnitude of the similarity (i.e. encourage the features to be orthogonal) then you can return the average absolute value of cosine similarity.
It seems like what you've implemented will attempt to maximize the similarity. But that doesn't appear to be in line with what you've stated. Also, to turn a minimization problem into an equivalent maximization problem you would usually just negate the measure. There's nothing wrong with a negative loss value. Taking the reciprocal of a strictly positive measure does convert it from minimization to a maximization problem, but also changes the behavior of the measure and probably isn't what you want.
Depending on what you actually want, one of these is likely to meet your needs
import torch.nn.functional as F

def loss_func(feat1, feat2):
    # minimize average magnitude of cosine similarity
    return F.cosine_similarity(feat1, feat2).abs().mean()

def loss_func(feat1, feat2):
    # minimize average cosine similarity
    return F.cosine_similarity(feat1, feat2).mean()

def loss_func(feat1, feat2):
    # maximize average magnitude of cosine similarity
    return -F.cosine_similarity(feat1, feat2).abs().mean()

def loss_func(feat1, feat2):
    # maximize average cosine similarity
    return -F.cosine_similarity(feat1, feat2).mean()


Instruction: Pass weights into CrossEntropyLoss in correct order

Input:
I'm trying to use weight in torch.nn.CrossEntropyLoss,
but I'm not sure which order should I put
e.g.
    weight = torch.tensor([1.0, 52337/34649, 52337/11066]).to(device)

    criterion = nn.CrossEntropyLoss(weight=weight)

My class0 has 52337 examples it's labeled as 0 in target value, so I take 1.0
My class1 has 34649 examples it's labeled as 2 in target value, so I take 52337/34649
My class2 has 34649 examples it's labeled as 1 in target value, so I take 52337/11066
but I'm not sure which order should put in weight array,
My question is
Is there a way to show what is class0, class1 and class2 in CrossEntropy?
or the CrossEntropy will figure out the weights by itself?

Output:
after you define your criterion as above you will have to call it like:
loss = criterion (inputs, targets)

the targets will be encoded 0 .. C-1 where C is number of classes (0,1 or 2 in your case). So your weights order and number should correspond to your targets.


Instruction: Pytorch transform.ToTensor() changes image

Input:
I want to convert images to tensor using torchvision.transforms.ToTensor(), after processing I printed the image but the image became so weird. Here is my code:
trans = transforms.Compose([
    transforms.ToTensor()])

demo = Image.open(img)
demo_img = trans(demo)
demo_array = demo_img.numpy()*255
print(Image.fromarray(demo_array.astype(np.uint8)))

The original image is this
But after processing it is showed like this
Did I write something wrong or miss something?

Output:
It seems that the problem is with the channel axis.
If you look at torchvision.transforms docs, especially on ToTensor()

Converts a PIL Image or numpy.ndarray (H x W x C) in the range
[0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0]

So once you perform the transformation and return to numpy.array your shape is: (C, H, W) and you should change the positions, you can do the following:
demo_array = np.moveaxis(demo_img.numpy()*255, 0, -1)

This will transform the array to shape (H, W, C) and then when you return to PIL and show it will be the same image.
So in total:
import numpy as np
from PIL import Image
from torchvision import transforms

trans = transforms.Compose([transforms.ToTensor()])

demo = Image.open(img)
demo_img = trans(demo)
demo_array = np.moveaxis(demo_img.numpy()*255, 0, -1)
print(Image.fromarray(demo_array.astype(np.uint8)))


Instruction: How can I add new layers on pre-trained model with PyTorch? (Keras example given.)

Input:
I am working with Keras and trying to analyze the effects on accuracy that models which are built with some layers with meaningful weights, and some layers with random initializations.
Keras:
I load VGG19 pre-trained model with include_top = False parameter on load method.
model = keras.applications.VGG19(include_top=False, weights="imagenet", input_shape=(img_width, img_height, 3))

PyTorch:
I load VGG19 pre-trained model until the same layer with the previous model which loaded with Keras.
model = torch.hub.load('pytorch/vision:v0.6.0', 'vgg19', pretrained=True)
new_base =  (list(model.children())[:-2])[0]

After loaded models following images shows summary of them. (Pytorch, Keras)

So far there is no problem. After that, I want to add a Flatten layer and a Fully connected layer on these pre-trained models. I did it with Keras but I couldn't with PyTorch.

The output of new_model.summary() is that:

My question is, How can I do add a new layer in PyTorch?

Output:
If all you want to do is to replace the classifier section, you can simply do so. That is :
model = torch.hub.load('pytorch/vision:v0.6.0', 'vgg19', pretrained=True)
model.classifier = nn.Linear(model.classifier[0].in_features, 4096)
print(model)

will give you:
Before:
VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (17): ReLU(inplace=True)
    (18): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (19): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (24): ReLU(inplace=True)
    (25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (26): ReLU(inplace=True)
    (27): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (31): ReLU(inplace=True)
    (32): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (33): ReLU(inplace=True)
    (34): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (35): ReLU(inplace=True)
    (36): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)

After:
VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (17): ReLU(inplace=True)
    (18): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (19): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (24): ReLU(inplace=True)
    (25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (26): ReLU(inplace=True)
    (27): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (31): ReLU(inplace=True)
    (32): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (33): ReLU(inplace=True)
    (34): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (35): ReLU(inplace=True)
    (36): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
  (classifier): Linear(in_features=25088, out_features=4096, bias=True)
)

Also note that when you want to alter an existing architecture, you have two phases. You first get the modules you want (that's what you have done there) and then you must wrap that in a nn.Sequential because your list does not implement a forward() and thus you cant really feed it anything. its just a collection of modules.
So you need to do something like this in general (as an example):
features = nn.ModuleList(your_model.children())[:-1]
model = nn.Sequential(*features)
# carry on with what other changes you want to perform on your model

Note that if you want to create a new model and you intend on using it like:
output = model(imgs)

You need to wrap your features and new layers in a second sequential. That is, do something like this:
features = nn.ModuleList(your_model.children())[:-1]
model_features = nn.Sequential(*features)
some_more_layers = nn.Sequential(Layer1,
                                 Layer2,
                                 ... )

model = nn.Sequential(model_features,
                      some_more_layers)
#
output = model(imgs)

otherwise you had to do something like :
features_output = model.features(imgs)
output = model.classifier(features_output)


Instruction: What is the difference in RobertaTokenizer() and from_pretrained() way of initialising RobertaTokenizer?

Input:
I am a newbie to huggingface transformers and facing the below issue in training a RobertaForMaskedLM LM from scratch:
First, I have trained and saved a ByteLevelBPETokenizer as follows:
tokenizer = ByteLevelBPETokenizer()
print('Saving tokenizer at:', training_file)
tokenizer.train(files=training_file, vocab_size=VOCAB_SIZE, min_frequency=2,
special_tokens=["<s>","<pad>","</s>","<unk>","<mask>"])
tokenizer.save_model(tokenizer_mdl_dir)

Then, trained RobertaForMaskedLM using this tokenizer by creating a RobertaTokenizer as follows:
roberta_tokenizer = RobertaTokenizer(tokenizer_mdl + "/vocab.json", tokenizer_mdl + "/merges.txt")

But now, when I try to test the trained LM using a fill-mask pipeline,
fill_mask_pipeline = pipeline("fill-mask", model=roberta_model, tokenizer=roberta_tokenizer)

I got the below error:

PipelineException: No mask_token () found on the input

So, I realized, the tokenizer that I have loaded, is tokenizing the <mask> token as well. But I couldn't understand why it is doing so. Please help me understand this.
After trying several things, I loaded the tokenizer differently,
roberta_tokenizer = RobertaTokenizer.from_pretrained(tokenizer_mdl)

And, now the fill_mask_pipeline runs without errors. So, what is the difference between loading a tokenizer using RobertaTokenizer() and using the .from_pretrained() method?

Output:
When you compare the property unique_no_split_tokens, you will see that this is initialized for the from_pretrained tokenizer but not for the other.
#from_pretrained
t1.unique_no_split_tokens
['</s>', '<mask>', '<pad>', '<s>', '<unk>']

#__init__
t2.unique_no_split_tokens
[]

This property is filled by _add_tokens() that is called by from_pretrained but not by __init__. I'm actually not sure if this is a bug or a feature. from_pretrained is the recommended method to initialize a tokenizer from a pretrained tokenizer and should therefore be used.


Instruction: Best Loss Function for Multi-Class Multi-Target Classification Problem

Input:
I have a classification problem and I don't know how to categorize this classification problem. As per my understanding,

A Multiclass classification problem is where you have multiple mutually exclusive classes and each data point in the dataset can only be labelled by one class. For example, in an Image Classification task for fruits, a fruit data point labelled as an apple cannot be an orange and an orange cannot be a banana and so on. Each data point, in this case can only be any one of the fruits of the fruits class and so is labelled accordingly.

Where as ...

A Multilabel classification is a problem where you have multiple sets of mutually exclusive classes of which the data point can be labelled simultaneously. For example, in an Image Classification task for Cars, a car data point labelled as a sedan cannot be a hatchback and a hatchback cannot be a SUV and so on for the type of car. At the same time, the same car data point can be labelled one from VW, Ford, Mercedes, etc. as the car manufacturer. So in this case, the car data point is labeled from two different  sets of mutually exclusive classes.

Please correct my understanding if I am thinking in a wrong way here.
Now to my problem, my classification problem with multiple classes, lets say A, B, C, D and E. Here each data point can have one or more than one classes from the set as shown below on the left:
|-------------|----------|              |-------------|-----------------|
|      X      |     y    |              |      X      |    One-Hot-Y    |
|-------------|----------|              |-------------|-----------------|
|     DP1     |   A, B   |              |     DP1     | [1, 1, 0, 0, 0] |
|-------------|----------|              |-------------|-----------------|
|     DP2     |   C      |              |     DP2     | [0, 0, 1, 0, 0] |
|-------------|----------|              |-------------|-----------------|
|     DP3     |   B, E   |              |     DP3     | [0, 1, 0, 0, 1] |
|-------------|----------|              |-------------|-----------------|
|     DP4     |   A, C   |              |     DP4     | [1, 0, 1, 0, 0] |
|-------------|----------|              |-------------|-----------------|
|     DP5     |   D      |              |     DP5     | [0, 0, 0, 1, 0] |
|-------------|----------|              |-------------|-----------------|

I One-Hot Encoded the labels for training as shown above on the right. My question is:

What Loss function (preferably in PyTorch) can I use for training the model to optimize for the One-Hot encoded output
What do we call such a classification problem? Multi-label or Multi-class?

Thank you for your answers!

Output:
What Loss function (preferably in PyTorch) can I use for training the
model to optimize for the One-Hot encoded output

You can use torch.nn.BCEWithLogitsLoss (or MultiLabelSoftMarginLoss as they are equivalent) and see how this one works out. This is standard approach, other possibility could be MultilabelMarginLoss.

What do we call such a classification problem? Multi-label or Multi-class?

It is multilabel (as multiple labels can be present simultaneously). In one-hot encoding:
[1, 1, 0, 0, 0], [0, 1, 0, 0, 1] - multilabel
[0, 0, 1, 0, 0] - multiclass
[1], [0] - binary (special case of multiclass)

multiclass cannot have more than one 1 as all other labels are mutually exclusive.


Instruction: Pytorch RuntimeError: expected scalar type Float but found Byte

Input:
I am working on the classic example with digits. I want to create a my first neural network that predict the labels of digit images {0,1,2,3,4,5,6,7,8,9}. So the first column of train.txt has the labels and all the other columns are the features of each label. I have defined a class to import my data:
class DigitDataset(Dataset):
    """Digit dataset."""

    def __init__(self, file_path, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.data = pd.read_csv(file_path, header = None, sep =" ")
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        labels = self.data.iloc[idx,0]
        images = self.data.iloc[idx,1:-1].values.astype(np.uint8).reshape((1,16,16))

        if self.transform is not None:
            sample = self.transform(sample)
        return images, labels

And then I am running these commands to split my dataset into batches, to define a model and a loss:
train_dataset = DigitDataset("train.txt")
train_loader = DataLoader(train_dataset, batch_size=64,
                        shuffle=True, num_workers=4)

# Model creation with neural net Sequential model
model=nn.Sequential(nn.Linear(256, 128), # 1 layer:- 256 input 128 o/p
                    nn.ReLU(),          # Defining Regular linear unit as activation
                    nn.Linear(128,64),  # 2 Layer:- 128 Input and 64 O/p
                    nn.Tanh(),          # Defining Regular linear unit as activation
                    nn.Linear(64,10),   # 3 Layer:- 64 Input and 10 O/P as (0-9)
                    nn.LogSoftmax(dim=1) # Defining the log softmax to find the probablities
for the last output unit
                  )

# defining the negative log-likelihood loss for calculating loss
criterion = nn.NLLLoss()

images, labels = next(iter(train_loader))
images = images.view(images.shape[0], -1)

logps = model(images) #log probabilities
loss = criterion(logps, labels) #calculate the NLL-loss

And I take the error:
---------------------------------------------------------------------------
   RuntimeError                              Traceback (most recent call last)
    <ipython-input-2-7f4160c1f086> in <module>
     47 images = images.view(images.shape[0], -1)
     48
---> 49 logps = model(images) #log probabilities
     50 loss = criterion(logps, labels) #calculate the NLL-loss

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self,
*input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self,
*input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

 ~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

 ~/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
-> 1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
   1692         output = input.matmul(weight.t())

RuntimeError: expected scalar type Float but found Byte

Do you know what is wrong? Thank you for your patience and help!

Output:
This line is the cause of your error:
images = self.data.iloc[idx, 1:-1].values.astype(np.uint8).reshape((1, 16, 16))

images are uint8 (byte) while the neural network needs inputs as floating point in order to calculate gradients (you can't calculate gradients for backprop using integers as those are not continuous and non-differentiable).
You can use torchvision.transforms.functional.to_tensor to convert the image into float and into [0, 1] like this:
import torchvision

images = torchvision.transforms.functional.to_tensor(
    self.data.iloc[idx, 1:-1].values.astype(np.uint8).reshape((1, 16, 16))
)

or simply divide by 255 to get values into [0, 1].


Instruction: word synonym / antonym detection

Input:
I need to create a classifier that takes 2 words and determines if they are synonyms or antonyms. I tried nltk's antsyn-net but it doesn't have enough data.
example:

capitalism <-[antonym]-> socialism
capitalism =[synonym]= free market
god <-[antonym]-> atheism
political correctness <-[antonym]-> free speach
advertising =[synonym]= marketing

I was thinking about taking a BERT model, because may be some of the relations would be embedded  in it and transfer-learn on a data-set that I found.

Output:
I would suggest a following pipeline:

Construct a training set from existing dataset of synonyms and antonyms (taken e.g. from the Wordnet thesaurus). You'll need to craft negative examples carefully.
Take a pretrained model such as BERT and fine-tune it on your tasks. If you choose BERT, it should be probably BertForNextSentencePrediction where you use your words/prhases instead of sentences, and predict 1 if they are synonyms and 0 if they are not; same for antonyms.


Instruction: Pytorch error when computing loss between two tensors. TypeError: __init__() takes 1 positional argument but 3 were given

Input:
When trying to compute the loss between two tensors rPPG = (shape(torch.Size([4, 128])) and BVP_label = (shape(torch.Size([4, 128]))) using the following function:
class Neg_Pearson(nn.Module):    # Pearson range [-1, 1] so if < 0, abs|loss| ; if >0, 1- loss
    def __init__(self):
        super(Neg_Pearson,self).__init__()
        return
    def forward(self, preds, labels):       # tensor [Batch, Temporal]
        loss = 0
        for i in range(preds.shape[0]):
            sum_x = torch.sum(preds[i])                # x
            sum_y = torch.sum(labels[i])               # y
            sum_xy = torch.sum(preds[i]*labels[i])        # xy
            sum_x2 = torch.sum(torch.pow(preds[i],2))  # x^2
            sum_y2 = torch.sum(torch.pow(labels[i],2)) # y^2
            N = preds.shape[1]
            pearson = (N*sum_xy - sum_x*sum_y)/(torch.sqrt((N*sum_x2 - torch.pow(sum_x,2))*(N*sum_y2 - torch.pow(sum_y,2))))

            print(N)
            #if (pearson>=0).data.cpu().numpy():    # torch.cuda.ByteTensor -->  numpy
            #    loss += 1 - pearson
            #else:
            #    loss += 1 - torch.abs(pearson)

            loss += 1 - pearson

        loss = loss/preds.shape[0]
        return loss

#3. Calculate the loss
loss_ecg = Neg_Pearson(rPPG, BVP_label)

I keep getting the following error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-f14cbf0fc84b> in <module>
      1 #3. Calculate the loss
----> 2 loss_ecg = Neg_Pearson(rPPG, BVP_label)

TypeError: __init__() takes 1 positional argument but 3 were given

I am new to Pytorch, I am not sure what is going on here. Any suggestions?

Output:
You have a typo there. instead try :
neg_pears_loss = Neg_Pearson()
loss = neg_pears_loss(rPPG, BVP_label)


Instruction: Install Pytorch for only inference time

Input:
I have already make some research but could not find any useful information. Is it possible to install something like mini pytorch just for load a pre-trained model and call prediction method?
Entire PyTorch library size is too big, so I would like to prevent on this issue.
Does anyone have idea? Thanks in advance.

Output:
For production you need to use libtorch and the whole package is around 160MB compressed I guess.
You ship the Dlls that your application requires and at the very minimum I guess it could be around 170-200 MBs if you only use the torch.dll
And there is not a pytorch mini or anything like that as far as I know.


Instruction: I can not import torch

Input:
when I tried the "import torch" ı am getting this error .How can ı fix it?
I left error as a photo
enter image description here

Output:
The directory shows Anaconda 3. Try to install pytorch using the Anaconda command in the below link
https://anaconda.org/pytorch/pytorch


Instruction: Why does the same PyTorch code (different implementation) give different loss?

Input:
I was tackling the Fashion MNIST data-set problem on Udacity. However my implementation of code is giving drastically different loss as compared to the solution shared by the Udacity team. I believe the only difference in my answer is the definition of the Neural Network and apart from that everything is the same. I am not able to figure out the reason for such a drastic difference in Loss.
Code 1: My solution:
import torch.nn as nn
from torch import optim

images, labels = next(iter(trainloader))
model = nn.Sequential(nn.Linear(784,256),
                 nn.ReLU(),
                 nn.Linear(256,128),
                 nn.ReLU(),
                 nn.Linear(128,64),
                 nn.ReLU(),
                 nn.Linear(64,10),
                 nn.LogSoftmax(dim=1))
# Flatten images
optimizer = optim.Adam(model.parameters(),lr=0.003)

criterion = nn.NLLLoss()

for i in range(10):
    running_loss = 0
    for images,labels in trainloader:
        images = images.view(images.shape[0], -1)

        output = model.forward(images)
        loss = criterion(output,labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss}")

# Loss is coming around 4000

Code 2: Official Solution:
from torch import nn, optim
import torch.nn.functional as F

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Linear(64, 10)

    def forward(self, x):
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.log_softmax(self.fc4(x), dim=1)

        return x

model = Classifier()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)
epochs = 5

for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        log_ps = model(images)
        loss = criterion(log_ps, labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss}")
# Loss is coming around 200

Is there any explanation for the vast difference in loss ?

Output:
You forgot to zero out/clear the gradients in your implementation. That is you are missing :
optimizer.zero_grad()

In other words simply do:
for i in range(10):
    running_loss = 0
    for images,labels in trainloader:
        images = images.view(images.shape[0], -1)

        output = model.forward(images)
        loss = criterion(output,labels)
        # missed this!
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss}")

and you are good to go!