Instruction: Training Detectron2 on part of COCO dataset

Input:
I'm trying to train model with Detectron2 and COCO dataset for vehicle and person detection and I'm having problems with model loading.
I've used posts here on SO and https://github.com/immersive-limit/coco-manager (filter.py file) code to filter COCO dataset to only include annotations and images from classes "person", "car", "bike", "truck" and "bicycle". Now my directory structure is:
main
  - annotations:
    - instances_train2017_filtered.json
    - instances_val2017_filtered.json
  - images:
    - train2017_filtered (lots of images inside)
    - val2017_filtered (lots of images inside)

Basically, the only thing that I've done here was to remove documents and images not corresponding to those classes, and changed their IDs (so they are from 1 to 5).
Then I've used code from Detectron2 tutorial:
import random

import cv2
from detectron2.data import MetadataCatalog, DatasetCatalog
from detectron2.data.datasets import register_coco_instances
from detectron2.engine import DefaultTrainer, DefaultPredictor
from detectron2.config import get_cfg
import os

from detectron2.model_zoo import model_zoo
from detectron2.utils.visualizer import Visualizer

register_coco_instances("train",
                        {},
                        "/home/jakub/Projects/coco/annotations/instances_train2017_filtered.json",
                        "/home/jakub/Projects/coco/images/train2017_filtered/")

register_coco_instances("val",
                        {},
                        "/home/jakub/Projects/coco/annotations/instances_val2017_filtered.json",
                        "/home/jakub/Projects/coco/images/val2017_filtered/")

metadata = MetadataCatalog.get("train")
dataset_dicts = DatasetCatalog.get("train")

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("train",)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 300
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 5

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.DATASETS.TEST = ("val", )
predictor = DefaultPredictor(cfg)

img = cv2.imread("demo/input.jpg")
outputs = predictor(img)

for d in random.sample(dataset_dicts, 1):
    im = cv2.imread(d["file_name"])
    outputs = predictor(im)
    v = Visualizer(im[:, :, ::-1],
                   metadata=metadata,
                   scale=0.8)
    out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    cv2.imwrite('demo/output_retrained.jpg', out.get_image()[:, :, ::-1])

During training, I get the following errors:
Unable to load 'roi_heads.box_predictor.cls_score.weight' to the model due to incompatible shapes: (81, 1024) in the checkpoint but (6, 1024) in the model!
Unable to load 'roi_heads.box_predictor.cls_score.bias' to the model due to incompatible shapes: (81,) in the checkpoint but (6,) in the model!
Unable to load 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (320, 1024) in the checkpoint but (20, 1024) in the model!
Unable to load 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (320,) in the checkpoint but (20,) in the model!
Unable to load 'roi_heads.mask_head.predictor.weight' to the model due to incompatible shapes: (80, 256, 1, 1) in the checkpoint but (5, 256, 1, 1) in the model!
Unable to load 'roi_heads.mask_head.predictor.bias' to the model due to incompatible shapes: (80,) in the checkpoint but (5,) in the model!

The model cannot predict anything useful after training, despite reducing total_loss during training. I get that I should get warnings because of size mismatch (I've reduced number of classes), which is normal from what I've seen on the internet, but I don't get "Skipped" after each error line. I think that model is actually not loading anything here, and I wonder why and how can I fix this.
EDIT
For comparison, a similar behaviour in almost identical situation was reported as an Issue, but it had "Skipped" at the end of each error line, making them effectively warnings, not errors:
https://github.com/facebookresearch/detectron2/issues/196

Output:
This 'warning' basically says that you are trying to initialize weights from a model that was trained on a different number of classes. This is expected as you have read.
I suspect you are not getting any results from your training because your MetadataCatalog does not have the 'thing_classes' property set. You are only calling
MetadataCatalog.get("train")

Calling
MetadataCatalog.get("train").set(thing_classes=["person", "car", "bike", "truck", "bicycle"])

Should solve the issue, but if it doesn't I'm pretty sure your json is corrupted.


Instruction: using 2d array as indices of a 4d array

Input:
I have a Numpy 2D array (4000,8000) from a tensor.max() operation, that stores the indices of the first dimension of a 4D array (30,4000,8000,3). I need to obtain a (4000,8000,3) array that uses the indices over this set of images and extract the pixels of each position in the 2D max array.
A = np.random.randint( 0, 29, (4000,8000), dtype=int)
B = np.random.randint(0,255,(30,4000,8000,3),dtype=np.uint8)

final = np.zeros((B.shape[1],B.shape[2],3))

r = 0
c = 0
for row in A:
  c = 0
  for col in row:
    x = A[r,c]
    final[r,c] = B[x,r,c]
    c=c+1
  r=r+1

print(final.shape)

Is there any vectorised way to do that? I am fighting with the RAM usage using loops. Thanks

Output:
You can use np.take_along_axis.
First let's create some data (you should have provided a reproducible example):
>>> N, H, W, C = 10, 20, 30, 3
>>> arr = np.random.randn(N, H, W, C)
>>> indices = np.random.randint(0, N, size=(H, W))

Then, we'll use np.take_along_axis. But for that the indices array must be of the same shape than the arr array. So we are using np.newaxis to insert axis where shapes don't match.
>>> res = np.take_along_axis(arr, indices[np.newaxis, ..., np.newaxis], axis=0)

It already gives usable output, but with a singleton dimension on first axis:
>>> res.shape
(1, 20, 30, 3)

So we can squeeze that:
>>> res = np.squeeze(res)

>>> res.shape
(20, 30, 3)

And eventually check if the data is as we wanted:
>>> np.all(res[0, 0] == arr[indices[0, 0], 0, 0])
True

>>> np.all(res[5, 3] == arr[indices[5, 3], 5, 3])
True


Instruction: Conv3d error on dimensions of a 5d tensor

Input:
I have a tensor shaped ([5, 1, 3, 126, 126]), which represents a video (5 frames each 126x126 rgb).
I need to forward it into a
self.resnet = nn.Sequential(
        nn.Conv3d(5,5,1),
        nn.UpsamplingBilinear2d(size=None, scale_factor=0.5)
    )

but i get
RuntimeError: Given groups=1, weight of size [5, 5, 1, 1, 1], expected input[5, 1, 3, 126, 126] to have 5 channels, but got 1 channels instead

I think that I have probably misunderstood how the conv3d works but I can't really understand why the expected dimensions are so different from the ones that my 5d tensor has at that moment

Output: The reason this is happening is because the shape of your tensor is wrong. The Conv3d class expects the batch_size to come first then the number of channels then the number of frames then the height and width. That is why you are getting the error. You should change the shape of your input tensor to [5,3,1,126,126] Your conv3d parameters are also wrong. The first number should be the number of input channels the conv3d is supposed to get which in your case is 3 because it is an rgb image. The second number is the number of output channels which you do not need to change.


Instruction: Problems initializing model in pytorch

Input:
I can't initialize my model in pytorch and get:
TypeError                                 Traceback (most recent call last)
<ipython-input-82-9bfee30a439d> in <module>()
288 dataset = News_Dataset(true_path=args.true_news_file,
                            fake_path=args.fake_news_file,
289                         embeddings_path=args.embeddings_file)
 --> 290 classifier = News_classifier_resnet_based().cuda()
291 try:
292   classifier.load_state_dict(torch.load(args.model_state_file))

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in
__call__(self, *input, **kwargs)
548             result = self._slow_forward(*input, **kwargs)
549         else:
--> 550             result = self.forward(*input, **kwargs)
551         for hook in self._forward_hooks.values():
552             hook_result = hook(self, input, result)

TypeError: forward() missing 1 required positional argument: 'input'

Someone asked for code. It is given below

class News_classifier_resnet_based(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.activation = torch.nn.ReLU6()
        self.sigmoid = torch.nn.Sigmoid()

        self.positional_encodings = PositionalEncoder()

        self.resnet = list(torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True).children())

        self.to_appropriate_shape = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=77)

        self.conv1 = torch.nn.Conv2d(in_channels=1,out_channels=64,kernel_size=7,stride=2,padding=3)
        self.conv1.weight = torch.nn.Parameter(self.resnet[0].weight[:,0,:,:].data)
        self.center = torch.nn.Sequential(*self.resnet[1:-2])
        self.conv2 = torch.nn.Conv2d(in_channels=512, out_channels=1, kernel_size=1)
        self.conv3 = torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=7)

        self.title_conv = torch.nn.Sequential(
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=3,stride=3),
                    self.activation(),
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=2,stride=2),
                    self.activation(),
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=2,stride=2)
                )
       self.title_lin = torch.nn.Linear(25,1)

       self.year_lin = torch.nn.Linear(10,1)
       self.month_lin = torch.nn.Linear(12,1)
       self.day_lin = torch.nn.Linear(31,1)
       self.date_lin = torch.nn.Linear(3,1)

       self.final_lin = torch.nn.Linear(3,1)

   def forward(self,x_in):
       #input shape - (batch_size, 3+title_len+seq_len, embedding_dim)
       #output shape - (batch_size, 1)
       year = x_in[:,0,:10]
       month = x_in[:,1,:12]
       day = x_in[:,2,:31]

       title = x_in[:,3:3+args.title_len,:]
       text = x_in[:,3+args.title_len:,:]
       title = self.positional_encodings(title)
       text = self.positional_encodings(text)

       text = text.unsqueeze(1)
       text = self.activation(self.to_appropriate_shape(text))
       text = self.activation(self.conv1(text))
       text = self.activation(self.center(text))
       text = self.activation(self.conv2(text))
       text = self.activation(self.conv3(text))
       text = text.reshape(args.batch_size,-1)

       title = title.unsqueeze(1)
       title = self.activation(self.title_conv(title))
       title = title.reshape(args.batch_size,-1)
       title = self.activation(self.title_lin(title))

       year = self.activation(self.year_lin(year))
       month = self.activation(self.month_lin(month))
       day = self.activation(self.day_lin(day))
       date = torch.cat([year,month,day], dim=-1)
       date = self.activation(self.date_lin(date))

       final = torch.cat([date,title,text], dim=-1)
       final = self.sigmoid(self.final_lin(final))

       return final

classifier = News_classifier_resnet_based().cuda()

What should I do? StackOverflow asked for more details. I'm trying to classify texts using word embeddings but problem lies in last line. I am working in google colab. Also when I created some models in other code blocks, I've got no problems

Output:
The problem is in your init function. When you create title_conv, insted of passing the activation object previously created,  your are calling the activation without arguments. You can fix it by changing that part of code with this:
self.title_conv = torch.nn.Sequential(
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=3,stride=3),
                    self.activation, # Notice I have removed ()
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=2,stride=2),
                    self.activation, # Notice I have removed ()
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=2,stride=2)
                )


Instruction: MNIST dataset is failed to transform as tensor object

Input:
How to properly transform the MNIST dataset to tensor type? I tried below but not work. The error message AttributeError: 'int' object has no attribute 'type' indicates it is not tensor type.
The codes below can be tested in Google Colab.
It appears that PyTorch Version 1.3.1 can run this, but not for 1.5.1.
>>> import torch
>>> import torch.nn as nn
>>> import torchvision.transforms as transforms
>>> import torchvision.datasets as dsets
>>> import numpy as np
>>> torch.__version__
1.5.1+cu101

>>> train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor())
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz
100.1%Extracting ./data/MNIST/raw/train-images-idx3-ubyte.gz to ./data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./data/MNIST/raw/train-labels-idx1-ubyte.gz
113.5%Extracting ./data/MNIST/raw/train-labels-idx1-ubyte.gz to ./data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw/t10k-images-idx3-ubyte.gz
100.4%Extracting ./data/MNIST/raw/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz
180.4%Extracting ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw
Processing...
/pytorch/torch/csrc/utils/tensor_numpy.cpp:141: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.
Done!

>>> print("Print the training dataset:\n ", train_dataset)
Print the training dataset:
  Dataset MNIST
    Number of datapoints: 60000
    Root location: ./data
    Split: Train
    StandardTransform
Transform: ToTensor()

>>> print("Type of data element: ", train_dataset[0][1].type())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute 'type'

Output:
You need to access Ist element (corresponding to image tensors), not the second one (labels) i.e.
>>> print("Type of data element: ", train_dataset[0][0].type())
Type of data element:  torch.FloatTensor

>>> print(train_dataset[0][0].shape, train_dataset[0][1])
(torch.Size([1, 28, 28]), 5)


Instruction: Loss function negative log likelihood giving loss despite perfect accuracy

Input:
I am debugging a sequence-to-sequence model and purposely tried to perfectly overfit a small dataset of ~200 samples (sentence pairs of length between 5-50). I am using negative log-likelihood loss in pytorch. I get low loss (~1e^-5), but the accuracy on the same dataset is only 33%.
I trained the model on 3 samples as well and obtained 100% accuracy, yet during training I had loss. I was under the impression that negative log-likelihood only gives loss (loss is in the same region of ~1e^-5) if there is a mismatch between predicted and target label?
Is a bug in my code likely?

Output:
There is no bug in your code.
The way things usually work in deep nets is that the networks predicts the logits (i.e., log-likelihoods). These logits are then transformed to probability using soft-max (or a sigmoid function). Cross-entropy is finally evaluated based on the predicted probabilities.
The advantage of this approach is that is numerically stable, and easy to train with. On the other side, because of the soft-max you can never have "perfect" 0/1 probabilities for your predictions: That is, even when your network has perfect accuracy it will never assign probability 1 to the correct prediction, but "close to one". As a result, the loss will always be positive (albeit small).


Instruction: how can I install PyTorch?

Input:
I Install PyTorch in Anaconda But when I write :
pip install torchvision
In anaconda this Error show me :
No matching distribution found for torch==1.4.0 (from torchvision)
Did I install it badly?

Output:
The following worked for me. First install MKL:
conda install -c anaconda mkl

After this, install torchvision:
conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

For pip:
pip install pytorch torchvision


Instruction: Modify PyTorch model for inference - then resume training

Input:
I’d like to alternate inference and training in a model in PyTorch, but I need to modify it during inference, and I have some questions about that.
Can I unload the model from the gpu by calling model.to('cpu'), make a modified copy (and run it on the gpu), and then move the original back to gpu by calling model.to('gpu')? In other words, is moving the model gpu->cpu->gpu a lossless operation? What happens to the parameters that were passed to the optimizer? I don’t want to lose the optimizer state
What is the best way to make a copy of an in-memory model? I can save it and then reload a copy, but not sure if that is necessary just to copy.
If I want to run inference in half precision (more than 2x faster in this case), can I change the model to half and then change it back? Is that lossless? (Does the model keep a full-precision copy of everything, or does it replace weights with half-precision in place?)
The model is a ResNet50 lookalike.  Not enough GPU memory for two models :)

Output:
"What is the best way to make a copy of an in-memory model? I can save it and then reload a copy, but not sure if that is necessary just to copy."
You can copy a model with:
import copy
...
best_model = copy.deepcopy(model)

with best_model you can save to disk or load in other model, etc


Instruction: pytorch: Random classifier: ValueError: optimizer got an empty parameter list

Input:
Is there any best practice or efficient way to have a random classifier in pytorch? My random classifier basically looks like this:
def forward(self, inputs):
    # get a random tensor
    logits = torch.rand(batch_size, num_targets, num_classes)
    return logits

This should be fine in principle, but the optimizer raises a ValueError because the classifier - in contrast to all other classifiers / models in the system - does not have any parameters that can be optimized, obviously. Is there a torch built-in solution to this or must I change the system's architecture (to not perform optimization)?
Edit: If adding some arbitrary parameters to the model as shown below, the loss will raise an RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
def __init__(self, transformer_models: Dict, opt: Namespace):
    super(RandomMulti, self).__init__()
    self.num_classes = opt.polarities_dim
    # add some parameters so that the optimizer doesn't raise an exception
    self.some_params = nn.Linear(2, 2)

My assumption really would be that there is a simpler solution, since having a random baseline classifier is a rather common thing in machine learning.

Output:
Indeed having a "random" baseline is common practice, but usually you do not need to explicitly generate one, let alone "train" it. In most cases you can have quite accurate expectation values for the "random" baseline. For instance, in ImageNet classification you have 1000 categories of equal size, than predicting a category at random should give you an expected accuracy of 1/1000. You do not need to instantiate a random classifier to produce that number.
If you insist on explicitly instantiate a random classifier - what is the meaning of "training" it? There are the errors you get, pytorch simply cannot understand what you are doing. You can have a random classifier and you can evaluate its performance, but there is no meaning to training it.


Instruction: pytorch backward error, one of variables for gradient computation modified by an inplace operation

Input:
I'm new to pytorch, i've been trying to implement a text summarization network. When i call loss.backward() an error appears.

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10, 1, 1, 400]], which is output 0 of UnsqueezeBackward0, is at version 98; expected version 97 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

it's a seq2seq model, and i think the problem comes from this code snippet
    final_dists=torch.zeros((batch_size,dec_max_len,extended_vsize)) #to hold the model outputs with extended vocab
    attn_dists=torch.zeros((batch_size,dec_max_len,enc_max_len)) #to retain the attention weights over decoder steps
    coverages=torch.zeros((batch_size,dec_max_len,enc_max_len))  #the coverages are retained to compute coverage loss

    inp=self.emb_dropout(self.embedding(dec_batch[:,0])) #starting input: <SOS> shape [batch_size]
    #self.prev_coverage is the accumulated coverage

    coverage=None #initially none, but accumulates

    with torch.autograd.set_detect_anomaly(True):

        for i in range(1,dec_max_len):
            #NOTE: the outputs, atten_dists, p_gens assignments start from i=1 (DON'T FORGET!)
            vocab_dists,hidden,attn_dists_tmp,p_gen,coverage=self.decoder(inp,hidden,enc_outputs,enc_lens,coverage)

            attn_dists[:,i,:]=attn_dists_tmp.squeeze(1)
            coverages[:,i,:]=coverage.squeeze(1)
            #vocab_dists: [batch_size, 1, dec_vocab_size] Note: this is the normalized probability
            #hidden: [1,batch_size, dec_hid_dim]
            #attn_dists_tmp: [batch_size, 1, enc_max_len]
            #p_gen: [batch_size, 1]
            #coverage: [batch_size, 1, enc_max_len]

            #===================================================================
            #To compute the final dist in pointer-generator network by extending vocabulary
            vocab_dists_p=p_gen.unsqueeze(-1)*vocab_dists  #[batch_size,1,dec_vocab_size] note we want to maintain vocab_dists for teacher_forcing_ratio
            attn_dists_tmp=(1-p_gen).unsqueeze(-1)*attn_dists_tmp #[batch_size, 1, enc_max_len] note we want to maintain attn_dists for later use
            extra_zeros=torch.zeros((batch_size,1,max_art_oovs)).to(self.device)
            vocab_dists_extended=torch.cat((vocab_dists_p,extra_zeros),dim=2) #[batch_size, 1, extended_vsize]
            attn_dists_projected=torch.zeros((batch_size,1,extended_vsize)).to(self.device)
            indices=enc_batch_extend_vocab.clone().unsqueeze(1) #[batch_size, 1,enc_max_size]

            attn_dists_projected=attn_dists_projected.scatter(2,indices,attn_dists_tmp)
            #We need this otherwise we would modify a leaf Variable inplace
            #attn_dists_projected_clone=attn_dists_projected.clone()
            #attn_dists_projected_clone.scatter_(2,indices,attn_dists_tmp) #this will project the attention weights
            #attn_dists_projected.scatter_(2,indices,attn_dists_tmp)
            final_dists[:,i,:]=vocab_dists_extended.squeeze(1)+attn_dists_projected.squeeze(1)
            #===================================================================

            #teacher forcing, whether or not should use pred or dec sequence label
            if random.random()<teacher_forcing_ratio:
                inp=self.emb_dropout(self.embedding(dec_batch[:,i]))
            else:
                inp=self.emb_dropout(self.embedding(vocab_dists.squeeze(1).argmax(1)))

if i remove the for loop, and just do one step of updating attn_dists[:,1,:] etc, with toy loss from the outputs returned by forward, then it works fine.
Anyone has any idea what is wrong here? There is no inplace operation here. Many thanks!

Output:
From looking at your code, the problem likely comes from the following lines:
    attn_dists[:,i,:]=attn_dists_tmp.squeeze(1)
    coverages[:,i,:]=coverage.squeeze(1)

you are performing an in place operation that conflicts with the graph created by pytorch for backprop. It should be solved by concatenating the new info at every loop (you may run out of memory very soon!)
    attn_dists = torch.cat((attn_dists, attn_dists_tmp.squeeze(1)), dim=1)
    coverages = torch.cat(coverages, coverage.squeeze(1)),dim=1)

You should, change their initialization as well, otherwise you will endup of a tensor that is twice the size you were accounting for.


Instruction: How to initialise the Weights for specific task and Backpropagation modification

Input:
My model is used to predict values based on an minimising a loss function L. But, the loss function doesn’t have a single global minima value, but rather a large number of places where it achieves global minima.
So, the model is based like this:
Model Input is [nXn] tensor (let’s say: inp=[ [i_11, i_12, i_13, ..., i_1n],[i_21, i_22, ..., i_2n],...,[i_n1,i_n2, ..., i_nn] ]) and model output is [nX1] tensor (let’s say: out1=[o_1, o_2,..., o_n ])
Output tensor is out1 is passed in a function f to get out2 (let’s say: f(o_1, o_2, o_3,..., o_n)=[O_1, O_2, O_3, ..., O_n] )
These 2 values (i.e., out1 and out2) are minimised using MSELoss i.e., Loss = ||out1 - out2||
Now, there are a lot of values for [o_1, o_2, ..., o_n] for which the Loss goes to minimum.
But, I want the values of [o_1, o_2, ..., o_n] for which |o_1| + |o_2| + |o_3| + ... + |o_n| is maximum
Right now, the weights are initialised randomly:
self.weight = torch.nn.parameter.Parameter(torch.FloatTensor(in_features, out_features)) for some value of in_features and out_features

But by doing this, I am getting the values of [o_1, o_2, ..., o_n] for which |o_1| + |o_2| + |o_3| + ... + |o_n| is minimum.
I know this problem can be solved by without using deep-learning, but I am trying to get the results like this for some task computation.
Is there a way to change this to get the largest values predicted at the output of the neural net?
Or is there any other technique (backpropagation change) to change it to get the desired largest valued output?
Thanks in advance.
EDIT 1:
Based on the answer, out1=[o_1, o_2,..., o_n ] is tending to zero-valued tensor. In the initial epochs, out2=[O_1, O_2, O_3, ..., O_n] takes very large values, but subsequently comes down to lower values.
A snippet of code below will give the idea:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np

class Model(nn.Module):
    def __init__(self, inp_l, hid_l, out_l=1):
        super(Model, self).__init__()

        self.lay1 = nn.Linear(inp_l ,hid_l)
        self.lay2 = nn.Linear(hid_l ,out_l)
        self.dp = nn.Dropout(p=0.5)

    def forward(self, inp):

        self.out1= torch.tensor([]).float()

        for row in range(x.shape[0]):
            y = self.lay1(inp[row])
            y = F.relu(y)
            y = self.dp(y.float())
            y = self.lay2(y)
            y = F.relu(y)
            self.out1= torch.cat((self.out1, y))

        return self.out1.view(inp.shape[0],-1)

def function_f(inp, out1):
    '''
    Some functional computation is done to return out2.
    '''
    return out2

def train_model(epoch):
    model.train()
    t = time.time()
    optimizer.zero_grad()
    out1 = model(inp)

    out2 = function_f(inp, out1)

    loss1 = ((out1-out2)**2).mean()
    loss2 = -out1.abs().mean()
    loss_train = loss1 + loss2

    loss_train.backward(retain_graph=True)

    optimizer.step()

    if epoch%40==0:
        print('Epoch: {:04d}'.format(epoch+1),
                  'loss_train: {:.4f}'.format(loss_train.item()),
                  'time: {:.4f}s'.format(time.time() - t))

model= Model(inp_l=10, hid_l=5, out_l=1)
optimizer = optim.Adam(model.parameters(), lr=0.001)

inp = torch.randint(100, (10, 10))

for ep in range(100):
       train_model(ep)

But, out1 value goes to trivial solution i.e., zero-valued tensor which is the minimum valued solution. As mentioned before EDIT, I want to get the max-valued solution.
Thank you.

Output:
I am not sure I understand what you want.
Your weight initialization is overly complicated as well, you may just do:
self.weight = torch.nn.Linear(in_features, out_featues)

If you want to have the largest value of a batch of inputs you may simply do:
    y = self.weight(x)
    return y.max(dim=0)[0]

But I am not entirely sure that is what you meant with your question.
EDIT:
It seems you have two objectives. The first thing I would try is to convert both of them in losses to be minimized by the optimizer.
    loss1 = MSE(out1, out2)
    loss2 = - out1.abs().mean()
    loss = loss1 + loss2

minimizing loss will simutaneously minimize the MSE between out1 and out2 and maximize the absolute values of out1. (minimizing - out1.abs().mean() is the same as maximizing out1.abs().mean()).
Notice that it is possible your neural net will just create large biases and zero the weights as a lazy solution for the objective. You may turn of biases to avoid the problem, but I would still expect some other training problems.


Instruction: Why did this CUDA error occured in PyTorch?

Input:
On the process in building a RNN model, I met with the below error. The following are part of my code:
class RNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(emb_num, emb_size)
        self.dropout1 = nn.Dropout(dropout_rate)
        self.LSTM = nn.LSTM(50, 128, 1, bidirectional = True)
        self.dropout2 = nn.Dropout(dropout_rate)
        self.full_connect = nn.Linear(256 , 5) #  biLSTM state * 2
    def forward(self, x):
        x = self.embedding(x)
        x = x.permute(1,0,2)
        x = self.dropout1(x)
        _, (hn, cn) = self.LSTM(x)
        out = self.dropout2(hn)
        #print(out.shape)
        out = torch.cat([out[i, :, :] for i in range(2)], 1)
        out = out.squeeze()
        out = self.full_connect(out)
        return out

def train():
    optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate, weight_decay = 0.001)
    Loss = nn.CrossEntropyLoss()

    for epoch in range(epochs):
        model.train()
        max_acc = 0
        print('epoch:{}'.format(epoch + 1))
        for i, data in enumerate(trainloader, 0):
            X_train, y_train = data
            optimizer.zero_grad()
            X_train = X_train.long().to(device)
            y_train = y_train.long().to(device)
            output = model(X_train)
            loss = Loss(output, y_train)
            loss.backward()
            optimizer.step()
        print('loss:{:3f}'.format(loss))
        model.eval()
        acc = valid(validloader)
        print('epoch:{} acc:{}'.format(epoch+1, acc))
        if epoch + 1 == 50:
            torch.save(model.state_dict(), 'epoch50.pt')
        if acc > max_acc:
            max_acc = acc
            torch.save(model.state_dict(), 'max_acc model.pt')
    torch.save(model.state_dict(), 'final model.pt')

def valid(dataloader):
    correct = 0
    total = 0
    with torch.no_grad():
        for i, data in enumerate(dataloader, 0):
            X_train, y_train = data
            #optimizer.zero_grad()
            X_train = X_train.long().to(device)
            y_train = y_train.long().to(device)
            output = model(X_train)
            #loss = Loss(output, y_train)
            #loss.backward()
            #optimizer.step()
            correct += (torch.argmax(output, dim = 1) == y_train).sum().item()
            total += y_train.shape[0]
    return correct / total

In the above code, I created a devset to test the model in training. But after 4 epoch or more, this error occured:
Traceback (most recent call last):
  File "c:\Users\hhhh\Desktop\NLP-beginner\task2\task2.py", line 287, in <module>
    train()
  File "c:\Users\hhhh\Desktop\NLP-beginner\task2\task2.py", line 185, in train
    acc = valid(validloader)
  File "c:\Users\hhhh\Desktop\NLP-beginner\task2\task2.py", line 207, in valid
    correct += (torch.argmax(output, dim = 1) == y_train).sum().item()
RuntimeError: CUDA error: unspecified launch failure

I have tried to switch to cpu device to train the model, yet the training speed is to slow even for 1 epoch. Is it because my computer configuration that not enough to run this?

Output:
To check whether your system has CUDA or not:
from torch.cuda import is_available

def main():
    use_cuda = not args.no_cuda and is_available()
    dev = device("cuda" if use_cuda else "cpu")
    model = RNN().to(device=dev)
    # Call train and test methods below

if __name__ == '__main__':
    main()


Instruction: import matplotlib failed while deploying my model in AWS sagemaker

Input:
I have deployed my AWS model successfully.
but while testing i am getting runtime Error: "import matplotlib.pyplot as plt" . I think it is due to pytorch framework version i used(framework_version=1.2.0). I am facing the same issue when i use higher versions as well.
PyTorchModel(model_data=model_artifact,
                 role = role,
                 framework_version=1.2.0,
                 entry_point='predict.py',
                 predictor_cls=ImagePredictor)

I have other issue when i use version=1.0.0. i.e i am not able to import libraries from sub directories and deployment itself is failing.
Eg: i have some code files in "Code" directory.
 from Code.CTModel import NetWork  ---> **this line will fail as "No module named Code" when i use version=1.0.0**

Ultimately i want to how to use/import libraries which are written under sub-directories.

Output:
It sounds like you want to inject some additional code libraries into the SageMaker PyTorch serving container. You might have to dig into the source code for how the PyTorch serving container is built to further customize it: https://github.com/aws/sagemaker-pytorch-inference-toolkit, or build your own image.
Digging into that source code a bit, I see that the container has enabled the importing of arbitrary code, but only when "multi-model mode" is enabled. Can you verify that the code exists under a directory "code" in your model directory and that "multi-model mode" is enabled?
    def initialize(self, context):
        # Adding the 'code' directory path to sys.path to allow importing user modules when multi-model mode is enabled.
        if (not self._initialized) and ENABLE_MULTI_MODEL:
            code_dir = os.path.join(context.system_properties.get("model_dir"), 'code')
            sys.path.append(code_dir)
            self._initialized = True

Reference: https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/c4e7abc49aeebc2f9b6035337548a90e4330113d/src/sagemaker_pytorch_serving_container/handler_service.py#L47
If this all seems complicated to you (it is), you might want to look into some standardized formats for serializing your PyTorch model such as https://onnx.ai/. I'd love to learn more about what you're trying to do here sometime if you reach out to me at contact@modelzoo.dev. I'm beta-testing a platform that enables deployment in a single line of code and would love to test it out here.


Instruction: How does one save torch.nn.Sequential models in pytorch properly?

Input:
I am very well aware of loading the dictionary and then having a instance of be loaded with the old dictionary of parameters (e.g. this great question & answer). Unfortunately, when I have a torch.nn.Sequential I of course do not have a class definition for it.
So I wanted to double check, what is the proper way to do it. I believe torch.save is sufficient (so far my code has not collapsed), though these things can be more subtle than one might expect (e.g. I get a warning when I use pickle but torch.save uses it internally so it's confusing). Also, numpy has it's own save functions (e.g. see this answer) which tend to be more efficient, so there might be a subtle trade off I might be overlooking.

My test code:

# creating data and running through a nn and saving it

import torch
import torch.nn as nn

from pathlib import Path
from collections import OrderedDict

import numpy as np

import pickle

path = Path('~/data/tmp/').expanduser()
path.mkdir(parents=True, exist_ok=True)

num_samples = 3
Din, Dout = 1, 1
lb, ub = -1, 1

x = torch.torch.distributions.Uniform(low=lb, high=ub).sample((num_samples, Din))

f = nn.Sequential(OrderedDict([
    ('f1', nn.Linear(Din,Dout)),
    ('out', nn.SELU())
]))
y = f(x)

# save data torch to numpy
x_np, y_np = x.detach().cpu().numpy(), y.detach().cpu().numpy()
np.savez(path / 'db', x=x_np, y=y_np)

print(x_np)
# save model
with open('db_saving_seq', 'wb') as file:
    pickle.dump({'f': f}, file)

# load model
with open('db_saving_seq', 'rb') as file:
    db = pickle.load(file)
    f2 = db['f']

# test that it outputs the right thing
y2 = f2(x)

y_eq_y2 = y == y2
print(y_eq_y2)

db2 = {'f': f, 'x': x, 'y': y}
torch.save(db2, path / 'db_f_x_y')

print('Done')

db3 = torch.load(path / 'db_f_x_y')
f3 = db3['f']
x3 = db3['x']
y3 = db3['y']
yy3 = f3(x3)

y_eq_y3 = y == y3
print(y_eq_y3)

y_eq_yy3 = y == yy3
print(y_eq_yy3)

Related:

related question from forum: https://discuss.pytorch.org/t/how-to-save-nn-sequential-as-a-model/89117/14

Output:
As can be seen in the code torch.nn.Sequential is based on torch.nn.Module:
https://pytorch.org/docs/stable/_modules/torch/nn/modules/container.html#Sequential
So you can use
f = torch.nn.Sequential(...)
torch.save(f.state_dict(), path)

just like with any other torch.nn.Module.


Instruction: What should __len__ be for PyTorch when generating unlimited data?

Input:
Say I am trying to use PyTorch to learn the equation y = 2x and I want to generate an unlimited amount of data to train my model with. I am supposed to provide a __len__ function. Here's an example below. What should it be in this case? How do I specify the number of mini-batch iterations per epoch? Do I just set a number arbitrarily?
import numpy as np
from torch.utils.data import Dataset

class GenerateUnlimitedData(Dataset):
    def __init__(self):
        pass

    def __getitem__(self, index):
        x = np.random.randint(1,10)
        y = 2 * x
        return x, y

    def __len__(self):
        return 1000000 # This works but is not correct

Output:
You should use torch.utils.data.IterableDataset instead of torch.utils.data.Dataset. In your case it would be:
import torch

class Dataset(torch.utils.data.IterableDataset):
    def __init__(self, batch_size):
        super().__init__()
        self.batch_size = batch_size

    def __iter__(self):
        while True:
            x = torch.randint(1, 10, (self.batch_size,))
            y = 2 * x
            yield x, y

You should use batches (probably large ones), as that would speed up computations (pytorch is well suited for GPU computations on many samples at once).


Instruction: How to add new sample to CIFAR10 torchvision?

Input:
Hi I want to add my own images to the CIFAR10 dataset in torchvision, how can I do that?
train_data = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=train_transform)
train_data.add # or a workaround!

thanks

Output:
You can either create a custom dataset for CIFAR10, using the raw cifar10 images here or you can still use the CIFAR10 dataset inside your new custom dataset and then add your logic in the __getitem__() method.
This is a simple example to get you going :
class CIFAR10_2(torch.utils.data.Dataset):
    def __init__(self, dataset_path='/cifar10', transformations=None, should_download=True):
        self.dataset_train = torchvision.datasets.CIFAR10(dataset_path, download=should_download)
        self.transformations = transformations

    def __getitem__(self, index):
        # do as you wish , add your logic here
        (img, label) = self.dataset_train[index]
        # for transformations for example
        if self.transformations is not None:
            return self.transformations(img), label
        return img, label

    def __len__(self):
        return len(self.dataset_train)

you can get fancy and add logic for test,validation, etc and do what ever you like.


Instruction: Best way to save many tensors of different shapes?

Input:
I would like to store thousands to millions of tensors with different shapes to disk. The goal is to use them as a time series dataset. The dataset will probably not fit into memory and I will have to load samples or ranges of samples from disk.
What is the best way to accomplish this while keeping storage and access time low?

Output:
The easiest way to save anything in disk is by using pickle:
import pickle
import torch

a = torch.rand(3,4,5)

# save
with open('filename.pickle', 'wb') as handle:
    pickle.dump(a, handle)

# open
with open('filename.pickle', 'rb') as handle:
    b = pickle.load(handle)

You can also save things with pytorch directly, but that is just a pytorch wrapper around pikle.
import torch
x = torch.tensor([0, 1, 2, 3, 4])
torch.save(x, 'tensor.pt')

If you want to save multiple tensors in one file, you can wrap them in a dictionary:
import torch
x = torch.tensor([0, 1, 2, 3, 4])
a = torch.rand(2,3,4,5)
b = torch.zeros(37)
torch.save({"a": a, "b":b, "x", x}, 'tensors.pt')


Instruction: How to use a bert pretrained model somewhere else?

Input:
I followed this course https://www.coursera.org/learn/sentiment-analysis-bert about building a pretrained model for sentiment analysis. During the trining, at each epoch they saved the model using torch.save(model.state_dict(), f'BERT_ft_epoch{epoch}.model'). Now I want to use one of these models (the best one obviously) elsewhere, for example where a user can paste a tweet as an input and get the emotion of the writer. But I don't know how to load the model and predict, here's what I tried:
import torchvision.models as models
import torch

model = models.resnet101(pretrained=False)
model.load_state_dict(torch.load('Models/BERT_ft_epoch15.model'), strict=False)
model_ft.eval()
output = model_ft(input) #input is a tweets list

I get this error: TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not list

Output:
How to define, initialize, save and load models using Pytorch.
Initializing a model. That is done inheriting the class nn.Module, consider the simple two layer model:
import torch
import torch.nn as nn

class Model(nn.Module)
    def __init__(self, input_size=128, output_size=10):
        super(Model).__init__()

        self.layer1 = nn.Sequetial(nn.Linear(input_size, 64), nn.LeakyReLU())
        self.layer2 = nn.Linear(64, output_size)

    def forward(self, x):
        y = self.layer2(self.layer1(x))
        return y

The layers of the model are first initialized at __init__() and then we specify the operations of the forward pass in forward(). You can be creative there, just remember of using pytorch differenciable operations.
You initialize the model by creating an instance of the new class:
model = Model() # brand new instance!

After training your model you want to save it:
import torch
model = Model(128, 10) # initialization

torch.save(model.state_dict, 'model.pt') # saving state dict

You are not saving the model here, you are saving the state_dict this is an ordered dictionary that contains all the weights and biases and other parameters of your model. The reason we save the state_dict instead of the model directly can be found in the documentation (https://pytorch.org/tutorials/beginner/saving_loading_models.html). For now, just consider it best practice.
Finally, we arrive at how to load the model. You have to initialize the model first, then load the state_dict from disk.
model = Model(128, 10) # model initialization
model.load_state_dict('model.pt')
model.eval() # put the model in inference mode

Notice that, when we save the state_dict we may also save the optimizer and the graph used for back propagation. That is useful to checkpoint the training and resume it at a later stage.
    # in the training loop
    torch.save({"epoch": epoch,
                "model": model.state_dict,
                "optim": optim.state_dict,
                "loss": loss}, f'checkpoint{epoch}.pt')

I hope that paints a cleares picture for you =)


Instruction: pytorch training loss invariant with varying forward pass implementations

Input:
The following code (MNIST MLP in PyTorch) delivers approximately the same training loss regardless of having the last layer in the forward pass returning:

F.log_softmax(x)
(x)

Option 1 is incorrect because I use criterion = nn.CrossEntropyLoss() and yet the results are almost identical. Am I missing anything?
import torch
import numpy as np
import time
from torchvision import datasets
import torchvision.transforms as transforms
# number of subprocesses to use for data loading
num_workers = 0
# how many samples per batch to load
batch_size = 20

# convert data to torch.FloatTensor
transform = transforms.ToTensor()

# choose the training and test datasets
train_data = datasets.MNIST(root='data', train=True,
                                   download=True, transform=transform)
test_data = datasets.MNIST(root='data', train=False,
                                  download=True, transform=transform)

# prepare data loaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
    num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size,
    num_workers=num_workers)

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # linear layer (784 -> 1 hidden node)
        self.fc1 = nn.Linear(28 * 28, 512)
        self.dropout1= nn.Dropout(p=0.2, inplace= False)
        self.fc2 = nn.Linear(512, 256)
        self.dropout2= nn.Dropout(p=0.2, inplace= False)
        self.dropout = nn.Dropout(p=0.2, inplace= False)
        self.fc3 = nn.Linear(256, 10)

    def forward(self, x):
        # flatten image input
        x = x.view(-1, 28 * 28)
        # add hidden layer, with relu activation function
        x = F.relu(self.fc1(x))
        x = self.dropout1(x)
        x = F.relu(self.fc2(x))
        x = self.dropout2(x)
        x = self.fc3(x)
#        return F.log_softmax(x)
        return x

# initialize the NN
model = Net()
print(model)
model.to('cuda')
criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
n_epochs = 10

model.train() # prep model for training

for epoch in range(n_epochs):
    # monitor training loss
    train_loss = 0.0

    start = time.time()
    for data, target in train_loader:
        data, target = data.to('cuda'), target.to('cuda')

        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()*data.size(0)

    train_loss = train_loss/len(train_loader.dataset)

    print('Epoch: {} \tTraining Loss: {:.6f} \ttime: {:.6f}'.format(
        epoch+1,
        train_loss,
        time.time()-start
        ))

Output:
For numerical stability, the nn.CrossEntropyLoss() is implemented with the softmax layer inside it. So you should NOT use the softmax layer in your forward pass.
From the docs (https://pytorch.org/docs/stable/nn.html#crossentropyloss):

This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.

Using the softmax layer in the forward pass will lead to worse metrics because the gradient magnitudes are lowered (thus, the weight updates are also lowered). I learned it the hard way!
I guess your problem is that the loss is similar at the beginning of training, but at the end of the training, they should not. It is usually a good sanity check to overfit your model in one batch of data. The model should reach 100% accuracy if the batch is small enough. If the model is taking too long to train than you probably have a bug somewhere.
Hope that helps =)


Instruction: Received a make error while building warp-ctc pytorch

Input:
I am trying to build https://github.com/SeanNaren/warp-ctc.git on Google Colab, following this notebook. I am using these commands on Colab:
!git clone https://github.com/SeanNaren/warp-ctc.git;\
cd warp-ctc;\
mkdir build;\
cd build;\
cmake ..;\
make;

but I am receiving an error building it:
[-11%] Building NVCC (Device) object CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o
/content/drive/My Drive/simple_hwr/warp-ctc/src/ctc_entrypoint.cu(1): error: this declaration has no storage class or type specifier

/content/drive/My Drive/simple_hwr/warp-ctc/src/ctc_entrypoint.cu(1): error: expected a ";"

2 errors detected in the compilation of "/tmp/tmpxft_00000191_00000000-13_ctc_entrypoint.compute_70.cpp1.ii".
CMake Error at warpctc_generated_ctc_entrypoint.cu.o.cmake:279 (message):
  Error generating file /content/drive/My
  Drive/simple_hwr/warp-ctc/build/CMakeFiles/warpctc.dir/src/./warpctc_generated_ctc_entrypoint.cu.o

CMakeFiles/warpctc.dir/build.make:220: recipe for target 'CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o' failed
make[2]: *** [CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o] Error 1
CMakeFiles/Makefile2:146: recipe for target 'CMakeFiles/warpctc.dir/all' failed
make[1]: *** [CMakeFiles/warpctc.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

How can this be resolved?

Output:
This post documents an apparent fix for this issue, which also affects the original warp-ctc repository:

[...] the ctc_entrypoint.cu file needs to be a symlink. So, go to src dir and run:
rm ctc_entrypoint.cu
ln -s ctc_entrypoint.cpp ctc_entrypoint.cu

Then, re-run make, which should resolve the issue.


Instruction: How is batching normally performed for sequence data for an RNN/LSTM

Input:
This Udacity course notebook batches data in a way that is not intuitive to me.
For a long sequence of data, they first truncates the data so it can be evenly divided by batch_size. Next, they .reshape() the data to be (batch_size, -1). Then they creates a sliding window over these batches of sub-sequences. When the sliding window goes out of bounds, they add fake data (via wrap-around to) to the end.
This provided graphic might explain better than I can:

I'm just wondering if this practice is normal, or if there is a different way. It seems strange that the batches are non-consecutive sub-sequences. Wouldn't that make it hard to interpret the output of a single batch?
Is there a better approach? The woman in the video literally said something to the affect "I don't know why it's done this way, but I've seen it before and the network trains fine".

Output:
You should check the documentation on padded sequences from pytorch. (If I had more experience with it I would give you a more detailed explanation, but truth if that I never really understood them!)
Packed Sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.PackedSequence.html#torch.nn.utils.rnn.PackedSequence
Pack padded sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pack_padded_sequence.html#torch.nn.utils.rnn.pack_padded_sequence
Pad packed sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pad_packed_sequence.html#torch.nn.utils.rnn.pad_packed_sequence
Pad sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pad_sequence.html#torch.nn.utils.rnn.pad_sequence
Pack sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pack_sequence.html#torch.nn.utils.rnn.pack_sequence
The names are a bit confusing. But the idea is that you create a tensor with the size of your largest sequence in the batch. The other sequences will be padded to have the same size as the longest sequence in the bach. This packed padded sequence is given to the recurrent model (RNN, LTMS, GRU, your favorite). With that you can back arbitrary sequences with minor memory limitations.


Instruction: Flattening Efficientnet model

Input:
I am trying to remove the top layer in the efficientnet-pytorch implementation. However, if I simply replace the final _fc layer with my own fully connected layer, as suggested by the author in this github comment, I am worried that there is still a swish activation even after this layer, as opposed to having nothing as I expected. When I print the model, the final lines is as follows:
(_bn1): BatchNorm2d(1280, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
    (_avg_pooling): AdaptiveAvgPool2d(output_size=1)
    (_dropout): Dropout(p=0.2, inplace=False)
    (_fc): Sequential(
      (0): Linear(in_features=1280, out_features=512, bias=True)
      (1): ReLU()
      (2): Dropout(p=0.25, inplace=False)
      (3): Linear(in_features=512, out_features=128, bias=True)
      (4): ReLU()
      (5): Dropout(p=0.25, inplace=False)
      (6): Linear(in_features=128, out_features=1, bias=True)
    )
    (_swish): MemoryEfficientSwish()
  )
)

where _fc is my replaced module.
What I hoped to do was:
base_model = EfficientNet.from_pretrained('efficientnet-b3')
model = nn.Sequential(*list(base_model.children()[:-3]))

where in my mind base_model.children() flattens the model from the nested structure. However, now I cannot seem to be able to use the model as if I use a dummy input, x=torch.randn(1,3,255,255) I get the error: TypeError: forward() takes 1 positional argument but 2 were given.
It should be noted that model[:2](x) works, but not model[:3](x). model[2] seems to be the mobile blocks.
Here is a colab notebook with the above code.

Output:
This is a common misunderstanding of what print(net) actually does.
The fact that there is a _swish Module after the _fc simply means that for former was registered after the latter. You can check that in the code:
class EfficientNet(nn.Module):
    def __init__(self, blocks_args=None, global_params=None):

        # [...]

        # Final linear layer
        self._avg_pooling = nn.AdaptiveAvgPool2d(1)
        self._dropout = nn.Dropout(self._global_params.dropout_rate)
        self._fc = nn.Linear(out_channels, self._global_params.num_classes)
        self._swish = MemoryEfficientSwish()

The order in which they are defined is the order that they will be printed. When it comes to what exactly is performed, you have to check the forward:
def forward(self, inputs):
    # Convolution layers
    x = self.extract_features(inputs)

    # Pooling and final linear layer
    x = self._avg_pooling(x)
    x = x.flatten(start_dim=1)
    x = self._dropout(x)
    x = self._fc(x)

    return x

and, as you can see, there is nothing after self._fc(x), which means no Swish will be applied.


Instruction: Does my for loop run parallely, if all the tensors involved in the loop are on the GPU?

Input:
I have a list of tensors and all of them are present on the GPU. I obtained this list by splitting one tensor on the GPU using torch.split. I want to get list of sums of the list of tensors I have. So, in simple terms, I want to get a list in which, the first element is sum of first tensor in the list, and so on. If I run a for loop for this, does it get parallelised? If not, is there a way to make it run parallely? I want to parallelize it since the list is pretty long, and the sum operation can be done parallely, and independently on every tensor present on the list. If this operation can be performed on the GPU, the performance gain would be immense.
UPDATE : Consider I have a list of tensors as follows :
ls
[tensor([[0.8469, 0.3712, 0.2956],
         [0.6548, 0.5284, 0.8682],
         [0.5748, 0.2390, 0.1402],
         [0.0010, 0.1794, 0.6048],
         [0.4636, 0.4101, 0.6543]], device='cuda:0'),
 tensor([[0.2138, 0.3613, 0.8712],
         [0.4689, 0.0503, 0.7342],
         [0.1368, 0.0688, 0.9223]], device='cuda:0'),
 tensor([[0.3131, 0.6142, 0.1555],
         [0.4099, 0.5000, 0.7578],
         [0.7353, 0.2425, 0.4407],
         [0.5943, 0.0377, 0.4820],
         [0.5898, 0.9585, 0.6993]], device='cuda:0'),
 tensor([[0.8629, 0.3172, 0.4248],
         [0.9957, 0.6998, 0.0931],
         [0.0258, 0.9898, 0.5250]], device='cuda:0'),
 tensor([[0.0298, 0.4033, 0.9465],
         [0.2763, 0.9412, 0.4873]], device='cuda:0')]

As you can see, I have a list of 5 tensors of different shapes. Each tensor has a shape of 3 in their first dimension. The shape is different because of 0th dimension. So, in this example, the shapes of the tensor in the list are [[5,3], [3, 3], [5, 3], [3, 3], [2,3]]. I want to get a list of tensors from this list as follows :
sums = [torch.sum(li, axis=0) for li in ls]
sums
[tensor([2.5412, 1.7280, 2.5632], device='cuda:0'),
 tensor([0.8195, 0.4804, 2.5277], device='cuda:0'),
 tensor([2.6424, 2.3528, 2.5352], device='cuda:0'),
 tensor([1.8844, 2.0068, 1.0429], device='cuda:0'),
 tensor([0.3062, 1.3445, 1.4338], device='cuda:0')]

So, as you can see, the first tensor in the list is sum of first tensor in the list ls along the dimension 0. The second tensor is sum of second tensor in list ls along the dimension 0 and so on.
To do this task, I'm currently using a for loop. which iteratively calculates the sums and appends it to sums list. However, this is very inefficient as my list of tensors is really big, of the order of 100K, and doing this in each iteration is super inefficient. I wanted to find out if there is any way to do this more efficiently.
The list ls of tensors is obtained by splitting a big tensor like this :
splitter = [5, 3, 5, 3, 2]

A = torch.rand(18, 3).cuda()

ls = torch.split(A, splitter)
ls
(tensor([[0.1969, 0.6113, 0.3563],
         [0.9180, 0.7759, 0.5953],
         [0.0279, 0.4014, 0.2268],
         [0.9026, 0.3821, 0.1498],
         [0.3630, 0.9144, 0.3277]], device='cuda:0'),
 tensor([[2.1312e-02, 5.2311e-01, 8.9177e-02],
         [4.7427e-01, 2.4503e-04, 1.2559e-01],
         [5.1641e-01, 9.1357e-01, 9.5637e-01]], device='cuda:0'),
 tensor([[0.3730, 0.4251, 0.9437],
         [0.5634, 0.3086, 0.5891],
         [0.5602, 0.0872, 0.2128],
         [0.7717, 0.1920, 0.3977],
         [0.5787, 0.3488, 0.7499]], device='cuda:0'),
 tensor([[0.9338, 0.4330, 0.8843],
         [0.5646, 0.0574, 0.8790],
         [0.4692, 0.5831, 0.9160]], device='cuda:0'),
 tensor([[0.9786, 0.5209, 0.9364],
         [0.4370, 0.4917, 0.3672]], device='cuda:0'))

So, if avoiding the for loop is not possible, do anyone have any ideas on summing the main tensor A, according to a splitter provided? So, for example, in the code above, the splitter is [5, 3, 5, 3, 2]. So, I want to obtain a tensor res from tensor A such that the first row of res is sum of first 5 rows of A(because splitter[0] = 5) along dim=0. The second row of res is sum of next 3 rows (row 5 to row 7) of A. And so on. Can I do this without using a for loop? Or can I parallelise this for loop since the opearation it is doing are independent of each other and are mutually exclusive and exhaustive.
I hope the added details are enough. If I need to add any further details to the question, please let me know. Thanks in advance :)

Output:
PyTorch runs GPU operations asynchronously (see docs).

When you call a function that uses the GPU, the operations are enqueued to the particular device

This means, your sum operations may run in parallel.
I have made a simple experiment to test this. If I am right, it proves that you don't need to worry about parallelism here.
import torch

A = torch.rand(100000, 32, device='cuda')
splits = torch.split(A, 4)

Your code:
%%timeit -r1 -n5
sums = [s.sum() for s in splits]
torch.cuda.synchronize()

# Output: 5 loops, best of 1: 374 ms per loop

Added synchronization after every sum operation:
%%timeit -r1 -n5
sums = [torch.cuda.synchronize() or s.sum() for s in splits]

# Output: 5 loops, best of 1: 897 ms per loop


Instruction: PyTorch: NVIDIA NGC image or Docker Hub image?

Input:
What are the differences between the official PyTorch image on Docker Hub and the PyTorch image on NVIDIA NGC?
The NGC page is more documented than the Docker Hub page, which has no description. But the NGC image is also heavier by a few gigabytes, and it seems to require a CUDA 10.2-compatible driver.
Is there any advantage in using the NVIDIA NGC image instead of the one from Docker Hub?

Output:
I'm not a docker expert.
To the best of my knowledge Nvidia puts a lot of effort to ship GPU-optimized containers such that running GPU pytorch on Nvidia GPU with Nvidia container should have best possible performance.
Therefore, if you are using Nvidia hardware you should expect better performance using NGC containers. The gap, though, might not be that significant.


Instruction: Why error FileNotFoundError: [Errno 2] No such file or directory: './ML/model.pth' occurs?

Input:
Error

l_1      |     from Ml import demo_model
ml_1      |   File "/app/Ml/demo.py", line 84, in
ml_1      |     model = torch.load('./ML/model.pth',  map_location='cpu')
ml_1      |   File "/usr/local/lib/python3.7/site-packages/torch/serialization.py", line 584, in load
ml_1      |     with _open_file_like(f, 'rb') as opened_file:
ml_1      |   File "/usr/local/lib/python3.7/site-packages/torch/serialization.py", line 234, in _open_file_like
ml_1      |     return _open_file(name_or_buffer, mode)
ml_1      |   File "/usr/local/lib/python3.7/site-packages/torch/serialization.py", line 215, in init
ml_1      |     super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: './ML/model.pth'

occurs when starting the docker container.
Dockerfile:
FROM python:3.7

WORKDIR /app

COPY ./server/requirements.txt /app

COPY ./server/Ml/model.pth /app/Ml/

COPY ./server/Ml/demo.py /app/Ml/

RUN python -m pip install --no-cache-dir -r requirements.txt

CMD gunicorn -b=0.0.0.0:8443 -w=4 project.app:app --timeout 2000

The line of code where the error occurs:
best_model = torch.load('./ML/model.pth',  map_location='cpu')

Output: It seems like you have a typo: the folders are Ml (small "l") while in the code it is ML (capital "L").


Instruction: Pytorch how to increase batch size

Input: I currently have a tensor of torch.Size([1, 3, 256, 224]) but I need it to be input shape [32, 3, 256, 224].  I am capturing data in real-time so dataloader doesn't seem to be a good option.  Is there any easy way to take 32 of size torch.Size([1, 3, 256, 224]) and combine them to create 1 tensor of size [32, 3, 256, 224]?

Output:
You are probable using jit model, and the batch size must be exact like the one the model was trained on.
t = torch.rand(1, 3, 256, 224)
t.size() # torch.Size([1, 3, 256, 224])
t2= t.expand(32, -1,-1,-1)
t2.size() # torch.Size([32, 3, 256, 224])

Expanding a tensor does not allocate new memory, but only creates a new view on the existing tensor, and you get what you need. Only the tensor stride was changed.


Instruction: How to convert PyTorch tensor to image and send it with flask?

Input:
I have a neural network in PyTorch which gives an image as a Tensor. I have converted it to a numpy array and then followed the explanation here to send that image to the html. The problem is that it's always black.
This is my code in the PyTorch:
def getLinearImage():
  with torch.no_grad():
      test_z = torch.randn(1, 100).to(device)
      generated = G(test_z)
      generated = generated.cpu()
      numpy = generated[0].view(64, 64).numpy()
      numpu = (numpy + 1) * 128
      return numpy

This is the code in the flask where arr is the returned value from getLinearImage()
def getImage(arr):
    img = Image.fromarray(arr.astype("uint8"))
    file_object = io.BytesIO()

    img.save(file_object, "PNG")
    file_object.seek(0)

    return send_file(file_object, mimetype="image/PNG")

If I open a static image and I send it to getImage() it works but won't work with the generated one. In the html I call it like:
<img src="/getLinearImage" alt="User Image" height="100px" width="100px">

Output:
Logically speaking, since the static image works, the error is somewhere in your getLinearImage code. I would suggest running things through using PDB (or a debugger of your choice) to figure out why it's not generated correctly.
That said, I create a variable in your code:
numpu = (numpy + 1) * 128

which you don't seem to use, since you return the other variable afterwards:
return numpy

Could that be your problem?
Also: I presume that when you created this, you saved the original image locally to ensure something gets generated in the first place?


Instruction: Loss Analysis of Deep Learning

Input:
I'm new to deep learning, and I have built a graph convolution network. I used 5 fold cross-validations. After plotting the mean train_loss (blue) and validate_loss (orange) together, I got this baby.
MSE loss
As you can see, from the curvilinear trend of validate_loss, it seems that my networks learn few things. (I guess data? GCN frameworks? learning rate?)
Could you guys specifically help me to figure out the bugs?
I would appreciate that! If you don't get my point, please let me know.
class Scorer(nn.Module):
    """
    Three conv_layers and two fc_layers with Dropout
    """
    def __init__(self):
        super(Scorer, self).__init__()
        self.conv_layer1 = GraphConvNet(5, 64)
        self.conv_layer2 = GraphConvNet(64, 128)
        self.conv_layer3 = GraphConvNet(128, 256) # (I have tried delete conv_layer3)
        self.fc_layer1 = nn.Linear(256, 128)
        self.drop_layer1 = nn.Dropout(0.5)
        self.fc_layer2 = nn.Linear(128, 64)
        self.drop_layer2 = nn.Dropout(0.5)
        self.out_layer = nn.Linear(64, 1)

    def forward(self, NormLap, feat):
        h = self.conv_layer1(NormLap, feat)
        h = F.leaky_relu(h)
        h = self.conv_layer2(NormLap, h)
        h = F.leaky_relu(h)
        h = self.conv_layer3(NormLap, h)
        h = F.leaky_relu(h)
        h = self.fc_layer1(h)
        h = self.drop_layer1(h)
        h = F.leaky_relu(h)
        h = self.fc_layer2(h)
        h = self.drop_layer2(h)
        h = F.leaky_relu(h)
        h = self.out_layer(h)
        h = F.leaky_relu(h)

Below is my networks and parameters:

    # parameter setting
    learning_rate = 0.001 # (I have tried 1e-1, 1e-2)
    weight_decay = 1e-3  # (I have tried 1e-4)
    epochs = 500
    batch_size = 50  # (I have tried 30)

    model = Scorer()
    loss_func = nn.MSELoss()
    optimizer = th.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)

Output:
This is pretty much what train and validation loss is supposed to do.  Loss goes down over time; that's what an optimizer is trying to do.  train_loss keeps going down after valid_loss levels out or plateaus, indicating the model starts overfitting after epoch ~100 or so.  Whether MSE 0.3 is good or bad for your application depends entirely on the application, but yes, the optimizer is optimizing just fine.
Please have a look at this resource for how to interpret loss curves: https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/
"from the curvilinear trend of validate_loss, it seems that my networks learn few things" - It would help if you explain (in a lot of detail :) WHY you think that, in order to get better answers. What do you expect to see other than what you see? I look at the same graph and it seems to me your network is learning to model the data and predict whatever you're trying to predict.


Instruction: Using pre-trained weights in PyTorch

Input:
I am working on implementing a research paper based on computer vision in PyTorch. I have built the model architecture by referring to the paper. The author has uploaded saved weights on GitHub in ".pth.tar" format. I want to put the same weights in my model so that I can skip training and optimization part and directly get output from the neural net.
The paper is Learning to see in the dark.
Model architecture is as follow:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv1d(32, 12, 1)
        .
        .
   def forward(self, x):
        x = F.relu(self.conv1(x))
        .
        .
        return x
net = Net()

And it is to be followed by importing trained weight from google drive/cloud storage and defining the function to put the trained weights in the net.
PS: Model architecture is exactly same for both

Output:
If you are using google colab
#mount drive onto google colab

from google.colab import drive
drive.mount('/content/gdrive')

Define the path of the weights
weights_path="/content/gdrive/My Drive/weights.pth"

Extract the tar file
!tar -xvf weights.pth.tar

Load the weights into the model net
net=torch.load(weights_path)


Instruction: Cuda out of memory in loop on second forward pass?

Input:
I have the following problem:
model = MyModel()
model.load_state_dict(checkpoint[weights])
model.train()
data, label = get_data() # just take one trainings example
data.cuda()
for i in range(10): # lets predict data 10 times
    output = model(data)
    print(i)

I can do one forward step (i is printed 0) but then I get a Cuda OOM error at output = model(data).
It works if I use with torch.no_grad() but I'd like to train in this forward loop and so I need the gradients later.
Do I somehow have to clear the earlier output or data so it doesn't take up my Vram?
output = model(data.clone()) # this doesn't fix the problem

for i in range(10):
  model.zero_grad() # doesn't work either
  output = model(data)

Output:
Ok you have to manually remove the output tensor.
It is kept although the loop starts over again:
This is fixable by
for i in range(10):
  model.zero_grad()
  output = model(data)
  del output


Instruction: pytorch sliding window with unfold & fold

Input:
I'm trying to use a simple ssd that was trained on 300x300 data with annotated bounding boxes. If I crop the images manually, it works correctly but with full size images it fails (obviously) due to resizing large images to 300x300 removes many visual features.
I figured the good old sliding window will work here, but I have some problems rebuilding the images with detection and must admit I'm a bit clueless on how to approach, What I have so far is:
at first, I tried this:
chips = F.unfold(img_t.data, kernel_size=300)

following some examples from stack overflow, but this gives me error Input Error: Only 4D input Tensors are supported (got 3D)
so following some more googling I found something that works:
        patch_w = 300
        patch_h = 300
        patches = img_t.data.unfold(0, 3, 3).unfold(1, patch_w, patch_h).unfold(2, patch_w, patch_h)
#Visualise small part:
        fig = plt.figure(figsize=(4, 4))
        fig.tight_layout()
        plt.subplots_adjust(left=0.1, bottom=0.1, right=0.9, top=0.9, wspace=0.01, hspace=0.01)
        for i in range(4):
            for j in range(4):
                inp = transp(patches[0][i][j])
                inp = np.array(inp)

                ax = fig.add_subplot(4, 4, ((i*4)+j)+1, xticks=[], yticks=[])
                plt.imshow(inp)
        plt.show()

I then feed the patches to my detector and it looks more or less ok, but there's no overlap (object can be cut into pieces and missed) and more importantly, I can't reverse the process with unfold without getting drowned in exceptions.
I'm not adamant on using the fold/unfold combination for the task as what I really want is to be able to feed large image into the network in a way that will preserve as much information as possible, mark down the detections and rebuild image with bounding boxes from the smaller patches.
What I came up with is this:
new_im = Image.new("RGB", (300*dims[1], 300*dims[0]))
idx = 0
for i in range(dims[0]):
    for j in range(dims[1]):
        new_im.paste(tiles[idx], (j*300, i*300))
        idx += 1
new_im.show()

which rebuilds the image back, but in a very artificial way, where the detector annotates image cropping and returns them as image list which I here rebuild which is both ugly and inefficient.
After a bit of fiddling, I got it to work, but there comes a peculiarity of pytorch.It adds overlapping parts of patches instead of averaging them (see image). How can I fix it? I realise normalisation won't do anything here since it would normalise the good pixels as well, so it needs to just average overlapping pixels.

Also, please note the image was cropped erroneously. Simple code to reproduce:
def fold_unfold(img_path):
    transt = transforms.Compose([transforms.ToTensor(),
    # transforms.Normalize(mean=[0.5,0.5, 0.5], std=[0.5, 0.5, 0.5])
        ])

    transp = transforms.Compose([
        # transforms.Normalize(mean=[0.5,0.5, 0.5], std=[0.5, 0.5, 0.5]),
        transforms.ToPILImage()
    ])
    img_t = transt(Image.open(img_path))
    img_t = img_t.unsqueeze(0)
    kernel = 300
    stride = 200
    img_shape = img_t.shape
    B, C, H, W = img_shape
    # number of pixels missing:
    pad_w = W % kernel
    pad_h = H % kernel
    # Padding the **INPUT** image with missing pixels:
    img_t = F.pad(input=img_t, pad=(pad_w//2, pad_w-pad_w//2, pad_h//2, pad_h-pad_h//2), mode='constant', value=0)
    img_shape = img_t.shape
    B, C, H, W = img_shape
    print("\n-----input shape: ", img_shape)

    patches = img_t.unfold(3, kernel, stride).unfold(2, kernel, stride).permute(0,1,2,3,5,4)

    print("\n-----patches shape:", patches.shape)
    # reshape output to match F.fold input
    patches = patches.contiguous().view(B, C, -1, kernel*kernel)
    print("\n", patches.shape) # [B, C, nb_patches_all, kernel_size*kernel_size]
    patches = patches.permute(0, 1, 3, 2)
    print("\n", patches.shape) # [B, C, kernel_size*kernel_size, nb_patches_all]
    patches = patches.contiguous().view(B, C*kernel*kernel, -1)
    print("\n", patches.shape) # [B, C*prod(kernel_size), L] as expected by Fold
    # https://pytorch.org/docs/stable/nn.html#torch.nn.Fold

    output = F.fold(patches, output_size=(H, W), kernel_size=kernel, stride=stride)
    # mask that mimics the original folding:
    recovery_mask = F.fold(torch.ones_like(patches), output_size=(H,W), kernel_size=kernel, stride=stride)
    output = output/recovery_mask

    print(output.shape) # [B, C, H, W]
    aspil = transp(output[0])
    aspil.show()

still, the image is cropped quite a lot so something is still wrong:

Finally, getting the cropping done Code updated to the working version
The problem is coming from the way pytorch does the unfolding. The unfold from tensor method doesn't zero pad automatically, but rather stops the cropping by cutting off the part that didn't fit. I solved it by zero padding the tensor before cropping it.

Output:
I'm posting that way so it's easier for others to find a solution if they encounter similar problem.
Key Highlights:

pytorch unfold will crop out part of the image that doesn't fit into the sliding window used. (ex. with 300x300 image and 100x100 window, nothing would get cropped, but with 290x290 image and same window the cropping will well... crop out the last 90 rows and columns of the original image. Solution is to zero pad the image preemptively to match size of the sliding window
the size of the input image will change after zero padding (no surprise here) but it's easy to forget about that when reconstructing the original image.
Ideally you may want to crop the image to original size in the end, but with sliding window approach it makes more sense for my application to keep the padding around the image so that the center of my detector can be applied to edges of the image too.
Unfolding: I couldn't find a practical difference between patches = img_t.unfold(3, kernel, stride).unfold(2, kernel, stride).permute(0,1,2,3,5,4) and
patches = img_t.unfold(2, kernel, stride).unfold(3, kernel, stride) so explanation on that would be welcome.
Image tensor must be reshaped a number of times before it is possible to fold it back into the original (padded!) image.
normalisation - not in the sense of image transform but rather to revert the effect of sliding window overlap. Another peculiarity of pytorch I found is the way it pastes tensors onto one another when folding overlapping patches. Instead of taking the average of overlap area, it adds them together. This can be reverted with form of overlap mask. This has an exact shape of the produced patches and value of 1 for each point. After folding, the value of each pixel/point will be equal to the number of stacked folds. Ultimately this is the denominator for averaging colors for the overlaps.

The code that ultimately worked for me:
import torch
from torchvision.transforms import transforms
import torch.nn.functional as F
from PIL import Image

img_path = 'filename.jpg'

def fold_unfold(img_path):
    transt = transforms.Compose([transforms.ToTensor(),
       # transforms.Normalize(mean=[0.5,0.5, 0.5], std=[0.5, 0.5, 0.5])
    ])

    transp = transforms.Compose([
        # transforms.Normalize(mean=[0.5,0.5, 0.5], std=[0.5, 0.5, 0.5]),
        transforms.ToPILImage()
    ])
    img_t = transt(Image.open(img_path))
    img_t = img_t.unsqueeze(0)
    kernel = 300
    stride = 200  # smaller than kernel will lead to overlap
    img_shape = img_t.shape
    B, C, H, W = img_shape  # Batch size, here 1, channels (3), height, width
    # number of pixels missing in each dimension:
    pad_w = W % kernel
    pad_h = H % kernel
    # Padding the **INPUT** image with missing pixels:
    img_t = F.pad(input=img_t, pad=(pad_w//2, pad_w-pad_w//2,
                                    pad_h//2, pad_h-pad_h//2), mode='constant', value=0)
    img_shape = img_t.shape
    # UPDATE the shape information to account for padding
    B, C, H, W = img_shape
    print("\n----- input shape: ", img_shape)

    patches = img_t.unfold(3, kernel, stride).unfold(2, kernel, stride).permute(0, 1, 2, 3, 5, 4)

    print("\n----- patches shape:", patches.shape)
    # reshape output to match F.fold input
    patches = patches.contiguous().view(B, C, -1, kernel*kernel)
    print("\n", patches.shape) # [B, C, nb_patches_all, kernel_size*kernel_size]
    patches = patches.permute(0, 1, 3, 2)
    print("\n", patches.shape) # [B, C, kernel_size*kernel_size, nb_patches_all]
    patches = patches.contiguous().view(B, C*kernel*kernel, -1)
    print("\n", patches.shape) # [B, C*prod(kernel_size), L] as expected by Fold
    # https://pytorch.org/docs/stable/nn.html#torch.nn.Fold

    output = F.fold(patches, output_size=(H, W),
                    kernel_size=kernel, stride=stride)
    # mask that mimics the original folding:
    recovery_mask = F.fold(torch.ones_like(patches), output_size=(
        H, W), kernel_size=kernel, stride=stride)
    output = output/recovery_mask

    print(output.shape)  # [B, C, H, W]
    aspil = transp(output[0])
    aspil.show()

fold_unfold(img_path)


Instruction: Update targets in classification images

Input:
Why are we updating targets in the implementation of bayesian cnn with mc dropout here?
https://github.com/sungyubkim/MCDO/blob/master/Bayesian_CNN_with_MCDO.ipynb?fbclid=IwAR18IMLcdUUp90TRoYodsJS7GW1smk-KGYovNpojn8LtRhDQckFI_gnpOYc
def update_target(target, original, update_rate):
        for target_param, param in zip(target.parameters(), original.parameters()):
            target_param.data.copy_((1.0 - update_rate) * target_param.data + update_rate*param.data)

Output:
The implementation you have referred to is a data parallel one.
Which means, the author intends to train multiple networks with the same architecture but different hyper-parameters.
Although in an unconventional way, this is what update_target does:
update_target(net_test, net, 0.001)

It updates the net_test with a lower learning rate compared to net, but with the exact same parameter changes applied to original net, that is actually being trained. Only the change scales is different.
I am assuming that this is found to be useful in terms of computational efficiency, since only one of the networks are actually being "trained" during main training phase:
outputs = net(inputs)
loss = CE(outputs, labels)
loss.backward()
optimizer.step()

One less forward pass and one less backprop per step.


Instruction: load and freeze one model and train others in PyTorch

Input:
I have a model A that including three submodels model1, model2, model3.
the model flow: model1 --> model2 --> model3
I have trained model1 in an independent project.
The question is how to use the pre-trained model1 when training the model A?
Now, I try to implement this as follow:
I load the checkpoint of model1 by `model1.load_state_dict(torch.load(model1.pth)) and then set the requires_grad of the model1’s parameters as False?
Is it right?

Output:
Yes, that is correct.
When you structure your model the way you explained, what you are doing is correct.
ModelA consists of three submodels - model1, models, model3
Then you load the weights of each individual model with model*.load_state_dict(torch.load(model*.pth))
Then make requires_grad=False for the model you want to freeze.
for param in model*.parameters():
    param.requires_grad = False

You can also freeze weights of particular layers by accessing the submodules, for example, if you have a layer named fc in model1, then you can freeze its weights by making model1.fc.weight.requres_grad = False.


Instruction: Getting different eigenvalues between using numpy.linalg.eigh() and torch.symeig()

Input:
I am trying to understand why am I getting different eigenvalues between using numpy.linalg.eigh() and torch.symeig().
An example is as below:
Code:
import numpy as np
import torch

arr_symmetric = np.array([[1.,2,3], [2,5,6], [3,6,9]])
arr_symmetric, arr_symmetric.dtype

Output:
(array([[1., 2., 3.],
        [2., 5., 6.],
        [3., 6., 9.]]), dtype('float64'))

Code:
tsr_symmetric = torch.tensor(arr_symmetric)
tsr_symmetric

Output:
tensor([[1., 2., 3.],
        [2., 5., 6.],
        [3., 6., 9.]], dtype=torch.float64)

Code:
w, v = np.linalg.eigh(arr_symmetric)
w, v

Output:
(array([4.05517871e-16, 6.99264746e-01, 1.43007353e+01]),
 array([[-9.48683298e-01,  1.77819106e-01, -2.61496397e-01],
        [ 2.22044605e-16, -8.26924214e-01, -5.62313386e-01],
        [ 3.16227766e-01,  5.33457318e-01, -7.84489190e-01]]))

Code:
e, v = torch.symeig(tsr_symmetric, eigenvectors=True)
e, v

Output:
(tensor([-2.6047e-16,  6.9926e-01,  1.4301e+01], dtype=torch.float64),
 tensor([[ 9.4868e-01, -1.7782e-01,  2.6150e-01],
         [ 8.6389e-16,  8.2692e-01,  5.6231e-01],
         [-3.1623e-01, -5.3346e-01,  7.8449e-01]], dtype=torch.float64))

As you can see one of the eigenvalues is different, ie. 4.05517871e-16 vs. -2.6047e-16
Why is this happening?

Output:
4.05517871e-16 is very close to zero so is -2.6047e-16. They are very very close by. You can verify the same as below because input = V.e.V^T where e is a diagonal matrix with eigen values in the diagonal.
import numpy as np
import torch

arr_symmetric = np.array([[1.,2,3], [2,5,6], [3,6,9]])

e, v = np.linalg.eigh(arr_symmetric)
print (np.dot(v, np.dot(np.diag(e), v.T)))
for i in range(3):
    print (np.dot(arr_symmetric, v[:,i].reshape(-1,1)), e[i]*v[:,i])

e, v = torch.symeig(torch.tensor(arr_symmetric), eigenvectors=True)
print (torch.matmul(v, torch.matmul(e.diag_embed(), v.transpose(-2, -1))))
for i in range(3):
    print (np.dot(arr_symmetric, v[:,i].reshape(-1,1)), e[i]*v[:,i])

Output:
[[1. 2. 3.]
 [2. 5. 6.]
 [3. 6. 9.]]
[[3.33066907e-16]
 [8.88178420e-16]
 [8.88178420e-16]] [-3.84708031e-16  9.00430554e-32  1.28236010e-16]
[[ 0.12434263]
 [-0.57823895]
 [ 0.3730279 ]] [ 0.12434263 -0.57823895  0.3730279 ]
[[ -3.73959074]
 [ -8.04149487]
 [-11.21877222]] [ -3.73959074  -8.04149487 -11.21877222]
tensor([[1.0000, 2.0000, 3.0000],
        [2.0000, 5.0000, 6.0000],
        [3.0000, 6.0000, 9.0000]], dtype=torch.float64)
[[-3.33066907e-16]
 [ 0.00000000e+00]
 [-8.88178420e-16]] tensor([-2.4710e-16, -2.2502e-31,  8.2368e-17], dtype=torch.float64)
[[-0.12434263]
 [ 0.57823895]
 [-0.3730279 ]] tensor([-0.1243,  0.5782, -0.3730], dtype=torch.float64)
[[ 3.73959074]
 [ 8.04149487]
 [11.21877222]] tensor([ 3.7396,  8.0415, 11.2188], dtype=torch.float64)


Instruction: What does it mean to assert an object in python?

Input:
We are getting an assertion error in this line "assert dataset". We are printing the dataset object and the value we got was this '<datasets.TextDataset object at 0x000002531F10E408>'. We are using 'python 3.7' in this code. Why are we getting assertion error on dataset object?
We are basically trying to run the code of AttnGAN (https://github.com/taoxugit/AttnGAN).
The error is happening on Line: 130 in 'code/main.py'.
Code
dataset = TextDataset(cfg.DATA_DIR, split_dir, base_size=cfg.TREE.BASE_SIZE, transform=image_transform)
print(dataset)
assert dataset
dataloader = torch.utils.data.DataLoader(dataset, batch_size=cfg.TRAIN.BATCH_SIZE, drop_last=True, shuffle=bshuffle, num_workers=int(cfg.WORKERS))
Output
Load from:  C:\Users\admin\Desktop\TextToImage\AttnGAN-master (1)\AttnGAN-master\code/../data/birds/captions.pickle
<datasets.TextDataset object at 0x000002531F10E408>
Traceback (most recent call last):
File "./code/main.py", line 131, in
assert dataset
AssertionError
PS C:\Users\admin\Desktop\TextToImage\AttnGAN-master (1)\AttnGAN-master>

Output:
In this case, assert dataset is a not-very-clear way of checking if the dataset is empty.  assert throws an exception if the expression (in this case the dataset object) evaluates to false.
https://docs.python.org/3/library/stdtypes.html "Truth Value Testing" says

By default, an object is considered true unless its class defines
either a __bool__() method that returns False or a __len__() method
that returns zero

Loking at the github repo, TextDataset does define __len__().  The logical conclusion is that the returned length of the dataset in your case (after it is loaded) is zero.
Try to look at where it is loading data from, try to make sure the data is there, and print the length before the assertion.  Bonus: try to figure out why the original loading doesn't throw an exception but succeeds and produces an empty dataset.


Instruction: How to use AMD GPU for fastai/pytorch?

Input:
I'm using a laptop which has Intel Corporation HD Graphics 5500 (rev 09), and AMD Radeon r5 m255 graphics card.
Does anyone know how to it set up for Deep Learning, specifically fastai/Pytorch?

Output:
Update 2:
Microsoft has release Pytorch_DML a few hours ago.
You can now install it (in windows or WSL) using pypi package:
pytorch-directml 1.8.0a0.dev211021
pip install pytorch-directml

So if you are on windows or using WSL, you can hop in and give this a try!
Update :
As of Pytorch 1.8 (March 04, 2021), AMD ROCm versions are made available from Pytorch's official website. You can now easily install them on Linux and Mac, the same way you used to install the CUDA/CPU versions.
Currently, the pip packages are being provided only. Also, the Mac and Windows platforms are still not supported (I haven't tested with WSL2 though!)
Old answer:
You need to install the ROCm version. The official AMD instructions on building Pytorch is here.
There was previously a wheel package for rocm, but it seems AMD doesn't distribute that anymore, and instead, you need to build PyTorch from the source as the guide which I linked to explains.
However, you may consult this page, to build the latest PyTorch version: The unofficial page of ROCm/PyTorch.


Instruction: using huggingface Trainer with distributed data parallel

Input:
To speed up performace I looked into pytorches DistributedDataParallel and tried to apply it to transformer Trainer.
The pytorch examples for DDP states that this should at least be faster:

DataParallel is single-process, multi-thread, and only works on a single machine, while DistributedDataParallel is multi-process and works for both single- and multi- machine training. DataParallel is usually slower than DistributedDataParallel even on a single machine due to GIL contention across threads, per-iteration replicated model, and additional overhead introduced by scattering inputs and gathering outputs.

My DataParallel trainer looks like this:
import os
from datetime import datetime
import sys
import torch
from transformers import Trainer, TrainingArguments, BertConfig

training_args = TrainingArguments(
        output_dir=os.path.join(path_storage, 'results', "mlm"),  # output directory
        num_train_epochs=1,  # total # of training epochs
        gradient_accumulation_steps=2,  # for accumulation over multiple steps
        per_device_train_batch_size=4,  # batch size per device during training
        per_device_eval_batch_size=4,  # batch size for evaluation
        logging_dir=os.path.join(path_storage, 'logs', "mlm"),  # directory for storing logs
        evaluate_during_training=False,
        max_steps=20,
    )

mlm_train_dataset = ProteinBertMaskedLMDataset(
        path_vocab, os.path.join(path_storage, "data", "uniparc", "uniparc_train_sorted.h5"),
)

mlm_config = BertConfig(
        vocab_size=mlm_train_dataset.tokenizer.vocab_size,
        max_position_embeddings=mlm_train_dataset.input_size
)
mlm_model = ProteinBertForMaskedLM(mlm_config)
trainer = Trainer(
   model=mlm_model,  # the instantiated  Transformers model to be trained
   args=training_args,  # training arguments, defined above
   train_dataset=mlm_train_dataset,  # training dataset
   data_collator=mlm_train_dataset.collate_fn,
)
print("build trainer with on device:", training_args.device, "with n gpus:", training_args.n_gpu)
start = datetime.now()
trainer.train()
print(f"finished in {datetime.now() - start} seconds")

The output:
build trainer with on device: cuda:0 with n gpus: 4
finished in 0:02:47.537038 seconds

My DistributedDataParallel trainer is build like this:
def create_transformer_trainer(rank, world_size, train_dataset, model):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    os.environ["RANK"] = str(rank)
    os.environ["WORLD_SIZE"] = str(world_size)

    training_args = TrainingArguments(
        output_dir=os.path.join(path_storage, 'results', "mlm"),  # output directory
        num_train_epochs=1,  # total # of training epochs
        gradient_accumulation_steps=2,  # for accumulation over multiple steps
        per_device_train_batch_size=4,  # batch size per device during training
        per_device_eval_batch_size=4,  # batch size for evaluation
        logging_dir=os.path.join(path_storage, 'logs', "mlm"),  # directory for storing logs
        local_rank=rank,
        max_steps=20,
    )

    trainer = Trainer(
        model=model,  # the instantiated  Transformers model to be trained
        args=training_args,  # training arguments, defined above
        train_dataset=train_dataset,  # training dataset
        data_collator=train_dataset.collate_fn,
    )
    print("build trainer with on device:", training_args.device, "with n gpus:", training_args.n_gpu)
    start = datetime.now()
    trainer.train()
    print(f"finished in {datetime.now() - start} seconds")

mlm_train_dataset = ProteinBertMaskedLMDataset(
    path_vocab, os.path.join(path_storage, "data", "uniparc", "uniparc_train_sorted.h5"))

mlm_config = BertConfig(
    vocab_size=mlm_train_dataset.tokenizer.vocab_size,
    max_position_embeddings=mlm_train_dataset.input_size
)
mlm_model = ProteinBertForMaskedLM(mlm_config)
torch.multiprocessing.spawn(create_transformer_trainer,
     args=(4, mlm_train_dataset, mlm_model),
     nprocs=4,
     join=True)

The output:
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
build trainer with on device: cuda:1 with n gpus: 1
build trainer with on device: cuda:2 with n gpus: 1
build trainer with on device: cuda:3 with n gpus: 1
build trainer with on device: cuda:0 with n gpus: 1
finished in 0:04:15.937331 seconds
finished in 0:04:16.899411 seconds
finished in 0:04:16.938141 seconds
finished in 0:04:17.391887 seconds

About the inital forking warning: What is exaclty forked and is this expected?
And about the resulting time: Is the trainer incorrectly used since it seemed to be a lot slower than the DataParallel approach?

Output:
Kind of late to the party but anyway. I'm going to leave this comment here to help anyone wondering if it is possible to keep the parallelism in the tokenizer.
According to this comment on github the FastTokenizers seem to be the issue.
Also according to another comment on gitmemory you shouldn't use the tokenizer before forking the process. (which basically means before iterating through your dataloader)
So the solution is to not use FastTokenizers before training/fine-tuning or use the normal Tokenizers.
Check the huggingface documentation to find out if you really need the FastTokenizer.


Instruction: How does pytorch compute derivatives for simple functions?

Input:
When we talk about the auto-differentiation in the pytorch, we are usually presented a graphical structures of tensors based on their formulas, and pytorch will compute the gradients by tracing down the graphical tree using chain rules. However, I want to know what will happen at the leaf nodes? Does pytorch hardcode a whole list of basic functions with their analytical derivatives, or does it compute the gradients using numerical methods? A quick example:
import torch

def f(x):
    return x ** 2
x = torch.tensor([1.0], requires_grad=True)
y = f(x)
y.backward()
print(x.grad) # 2.0

In this example, does pytorch compute the derivative by $$ (x^2)' = 2x = 2 * 1 = 2 $$, or does pytorch compute in a way similar to $$ (1.00001^2 - 1^2) / (1.000001 - 1) ~ 2 $$ ?
Thanks!

Output:
See this paper for exact answer, specifically section 2.1 or figure 2.
In short, PyTorch has a list of basic functions and the expression of their derivatives. So, what is done in your case (y =xx), is evaluating $$ y' = 2x $$.
The numerical method you mentioned is called numerical differentiation or finite differences, and it is an approximation of the derivative. But it is not what PyTorch does.


Instruction: pytorch Gradient with respect to 3D input

Input:
I want to construct sobolev network for 3D input regression
In TensorFlow, the gradients of neural network model can be computed using tf.gradient like:
dfdx,dfdy,dfdz = tf.gradients(pred,[x,y,z])

Let M be a torch neural network with 5 layers.
If X is a set of (x,y,z)  (3dim data)
and M.forward(X)  is a 1 dim output
How can I compute like gradient of M.forward(X) with respect to X?  Something like:
tf.gradient(M.forward(X),X)

Output:
If you want to compute gradient of this function for  example

y_i = 5*(x_i + 1)²
Create tensor of size 2x1 filled with 1's that requires gradient

x = torch.ones(2, requires_grad=True)

Simple linear equation with x tensor created
y = 5 * (x + 1) ** 2

Let take o as multiple dimension equation
o = 1/2 *sum(y_i)
in python
o = (1/2) * torch.sum(y)

you can compute grad with
o.backward()
x.grad

you can get more information here https://www.deeplearningwizard.com/deep_learning/practical_pytorch/pytorch_gradients/