Instruction: Weights of pre-trained BERT model not initialized

Input:
I am using the Language Interpretability Toolkit (LIT) to load and analyze a BERT model that I pre-trained on an NER task.
However, when I'm starting the LIT script with the path to my pre-trained model passed to it, it fails to initialize the weights and tells me:
    modeling_utils.py:648] loading weights file bert_remote/examples/token-classification/Data/Models/results_21_03_04_cleaned_annotations/04.03._8_16_5e-5_cleaned_annotations/04-03-2021 (15.22.23)/pytorch_model.bin
    modeling_utils.py:739] Weights of BertForTokenClassification not initialized from pretrained model: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']
    modeling_utils.py:745] Weights from pretrained model not used in BertForTokenClassification: ['bert.embeddings.position_ids']

It then simply uses the bert-base-german-cased version of BERT, which of course doesn't have my custom labels and thus fails to predict anything. I think it might have to do with PyTorch, but I can't find the error.
If relevant, here is how I load my dataset into CoNLL 2003 format (modification of the dataloader scripts found here):
    def __init__(self):

        # Read ConLL Test Files

        self._examples = []

        data_path = "lit_remote/lit_nlp/examples/datasets/NER_Data"
        with open(os.path.join(data_path, "test.txt"), "r", encoding="utf-8") as f:
            lines = f.readlines()

        for line in lines[:2000]:
            if line != "\n":
                token, label = line.split(" ")
                self._examples.append({
                    'token': token,
                    'label': label,
                })
            else:
                self._examples.append({
                    'token': "\n",
                    'label': "O"
                })

    def spec(self):
        return {
            'token': lit_types.Tokens(),
            'label': lit_types.SequenceTags(align="token"),
        }

And this is how I initialize the model and start the LIT server (modification of the simple_pytorch_demo.py script found here):
    def __init__(self, model_name_or_path):
        self.tokenizer = transformers.AutoTokenizer.from_pretrained(
            model_name_or_path)
        model_config = transformers.AutoConfig.from_pretrained(
            model_name_or_path,
            num_labels=15,  # FIXME CHANGE
            output_hidden_states=True,
            output_attentions=True,
        )
        # This is a just a regular PyTorch model.
        self.model = _from_pretrained(
            transformers.AutoModelForTokenClassification,
            model_name_or_path,
            config=model_config)
        self.model.eval()

## Some omitted snippets here

    def input_spec(self) -> lit_types.Spec:
        return {
            "token": lit_types.Tokens(),
            "label": lit_types.SequenceTags(align="token")
        }

    def output_spec(self) -> lit_types.Spec:
        return {
            "tokens": lit_types.Tokens(),
            "probas": lit_types.MulticlassPreds(parent="label", vocab=self.LABELS),
            "cls_emb": lit_types.Embeddings()

Output:
This actually seems to be expected behaviour. In the documentation of the GPT models the HuggingFace team writes:

This will issue a warning about some of the pretrained weights not being used and some weights being randomly initialized. That’s because we are throwing away the pretraining head of the BERT model to replace it with a classification head which is randomly initialized.

So it seems to not be a problem for the fine-tuning. In my use case described above it worked despite the warning as well.


Instruction: Conda - ModuleNotFoundError: No module named 'torch'

Input:
Steps to reproduce:
I am using Anaconda on Windows to set up environment for this repo.
conda create --name pytorch-yolo
Then I install all dependencies with conda install --file requirements.txt
Which returns
PackagesNotFoundError: The following packages are not available from current channels:
- torch[version='>=1.2']

So I install pytorch with conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
When I am trying to evaluate the model from the following repo: https://github.com/eriklindernoren/PyTorch-YOLOv3 with python3 test.py --weights_path weights/yolov3.weights inside conda environment, it returns the following error ModuleNotFoundError: No module named 'torch'
If I run a Jupyter lab or notebook, or even go with python inside conda terminal, torch is easily imported.
What I've already tried:
Almost everything from this thread: No module named "Torch"
Specifically, creating a new conda environment with python set to 3.8.2, installing torch to base and then to the environment.
I'm also using pyenv to set global python to 3.8.2, but that doesn't help at all.
conda list shows I have pytorch installed
Can't wrap my head around this issue.

Output:
You are probably using the wrong python binary. Can you try python test.py --weights_path weights/yolov3.weights?
I am not familiar with Windows terminal, but you can get the path to the binaries by using the where command (which for Linux):
(pytorch-yolo) C:\Users\RemiChauvenne>where python3
C:\Users\RemiChauvenne\AppData\Local\Microsoft\WindowsApps\python3.exe

(pytorch-yolo) C:\Users\RemiChauvenne>where python
C:\Users\RemiChauvenne\miniconda3\envs\pytorch-yolo\python.exe
C:\Users\RemiChauvenne\AppData\Local\Microsoft\WindowsApps\python.exe

We can see that python3 does not go to the python binary inside the conda environment, whereas python is correctly linked to pytorch-yolo.


Instruction: PyTorch error: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

Input:
I have a very simple example
import torch

if __name__ == "__main__":
    DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    m = torch.nn.Linear(20, 30).to(DEVICE)
    input = torch.randn(128, 20).to(DEVICE)
    output = m(input)
    print('output', output.size())
    exit()

and I get:
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    output = m(input)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

I'm using PyTorch 1.7.1. Any help would be greatly appreciated.
Thank you.
EDIT. The update of python -m torch.utils.collect_env is:
Collecting environment information...
PyTorch version: 1.8.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 11.1.0
CMake version: version 3.18.4

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: TITAN RTX
GPU 1: TITAN RTX
GPU 2: TITAN RTX
GPU 3: TITAN RTX
GPU 4: TITAN RTX
GPU 5: TITAN RTX
GPU 6: TITAN RTX
GPU 7: TITAN RTX

Nvidia driver version: 460.39
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.8.0
[pip3] torchaudio==0.8.0
[pip3] torchsummary==1.5.1
[conda] Could not collect

Output:
As described in your log, pytorch 1.8 is installed, not 1.7.1. Otherwise please send your log again, using the right python binary.
I faced the exact same issue, using 1.8. Downgrading to 1.7.1 solved it (as mentioned on huggingface transformers github issue).


Instruction: Optimize input instead of network in pytorch

Input:
I am trying to optimize the inputs in some required task and I don't want to update my network as its freezed. I have written a minimal example but its not working as z has the same value in all iterations.  I am sure that I am doing some silly mistake in this process. Any guidance is highly appreciated.
thanks.
import torch

z = torch.rand((1,6))
z.requires_grad_(True)
optimizer = torch.optim.SGD([z], lr= 0.1)

criteria = torch.nn.MSELoss()

for i in range(5):
    optimizer.zero_grad()
    print(z)
    loss = criteria(z, z+torch.rand(1))
    #print(loss)
    loss.backward()
    optimizer.step()

##output
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)

Output:
The gradients are always 0. This can be seen as follows:
import torch

z = torch.rand((1,6))
z.requires_grad_(True)
z.retain_grad()
optimizer = torch.optim.SGD([z], lr= 0.1)

criteria = torch.nn.MSELoss()

for i in range(5):
    optimizer.zero_grad()
    # print(z)
    loss = criteria(z, z+torch.rand(1))
    # print(loss)
    loss.backward()
    print(z.grad)
    optimizer.step()

tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
...

I don't have the mathematical proof, but it probably comes from the definition of the target z+torch.rand(1), that directly depends on the input z by a simple addition.


Instruction: How to get topk's values with its indices (2D)?

Input:
I have two 3D tensor and I want to use one's top k indices get another top k.
For example for the following tensor
a = torch.tensor([[[1], [2], [3]],
                  [[4], [5], [6]]])

b = torch.tensor([[[7,1], [8,2], [9,3]],
                  [[10,4],[11,5],[12,6]]])

pytorch's topk function will give me the following.
top_tensor, indices = torch.topk(a, 2, dim=1)

# top_tensor: tensor([[[3], [2]],
#                    [[6],  [5]]])

# indices: tensor([[[2], [1]],
#                 [[2],  [1]]])

But I want to use the result of a, map to b.
# use indices to do something for b, get torch.tensor([[[8,2], [9,3]],
#                                                      [[11,5],[12,6]]])

In this case, I don't know the real values of b, so I can't use topk to b.
on the other word, I want to get a funtion foo_slice as following:
top_tensor, indices = torch.topk(a, 2, dim=1)
# top_tensor == foo_slice(a, indices)

Is there any approach to achieve this using pytorch?
Thanks!

Output:
The solution what you are looking for is here
So the code based solution to your problem is as follows
#inputs are changed in order from the above ques

a = torch.tensor([[[1], [2], [3]],
                  [[5], [6], [4]]])

b = torch.tensor([[[7,1], [8,2], [9,3]],
                  [[11,5],[12,6],[10,4]]])

top_tensor, indices = torch.topk(a, 2, dim=1)

v = [indices.view(-1,2)[i] for i in range(0,indices.shape[1])]

new_tensor = []
for i,f in enumerate(v):
      new_tensor.append(torch.index_select(b[i], 0, f))
print(new_tensor ) #[tensor([[9, 3],
                   #         [8, 2]]),
                   #tensor([[12,  6],
                   #        [11,  5]])]


Instruction: Pip is uninstalling pytorch built from source when i use 'pip install '. Is there a way?

Input:
I needed to build pytorch from source in order to use the latest cuda toolkit - hence I did that. My environment is a conda enviornment.
The package I am trying to install is allennlp.
Pip list is not showing torch installed, conda is showing it is.
Tried building allennlp from source through:
conda skeleton pypi <package>
conda build <package>

conda build is also causing problems due to incompability with pip packages etc...
I am just wondering if there is a simpler way to do this?
For example: Tell pip that torch is already installed so it stops uninstalling the current torch (why can't it just pick it up as installed).
Appreciate the help!

Output: You can install allennlp with pip install --no-deps allennlp, but then you have to make sure the other dependencies are there yourself.


Instruction: Decoder upsample size

Input:
I have a simple autoencoder which looks like this:
class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()

        self.channels = [3, 8, 16, 32]

        self.encoder = nn.Sequential(
                # layer 1
                nn.Conv2d(self.channels[0], self.channels[1], 3, 1, 1),
                nn.BatchNorm2d(self.channels[1]),
                nn.ReLU(inplace=True),
                nn.MaxPool2d(kernel_size=4, stride=4),
                # layer 2
                nn.Conv2d(self.channels[1], self.channels[2], 3, 1, 1),
                nn.BatchNorm2d(self.channels[2]),
                nn.ReLU(inplace=True),
                nn.MaxPool2d(kernel_size=4, stride=4),
                # layer 3
                nn.Conv2d(self.channels[2], self.channels[3], 3, 1, 1),
                nn.BatchNorm2d(self.channels[3]),
                nn.ReLU(inplace=True),
                nn.MaxPool2d(kernel_size=2, stride=2)
            )

        self.decoder = nn.Sequential(
                nn.Conv2d(self.channels[3], self.channels[2], 3, 1, 1),
                nn.ReLU(inplace=True),
                nn.Upsample(size=(15, 20), mode='bilinear', align_corners=False),

                nn.Conv2d(self.channels[2], self.channels[1], 3, 1, 1),
                nn.ReLU(inplace=True),
                nn.Upsample(scale_factor=4, mode='bilinear', align_corners=False),

                nn.Conv2d(self.channels[1], 1, 1),
                nn.ReLU(inplace=True),
                nn.Upsample(scale_factor=4, mode='bilinear', align_corners=False)
            )

    def forward(self, x):
        h = self.encoder(x)
        y = self.decoder(h)
        return y

Can someone clarify if the nn.Upsample(size=(15, 20)... in the Decoder has something to do with restoring the images to their original dimensions? My input images(torch tensors) are of size 240*320 and the network is supposed to restore the input tensor to its original size.
Also if I add a fourth layer to my encoder with 64 channels and a 4th layer to my decoder (similar to the layers above), what should the upsampling size be in the first layere of the decode?

Output:
As mentioned in PyTorch Documentation, You may define an upsampling layer with a scale factor or an output size. Stating output_size=(H, W) will make sure the output size will be (H, W), regardless of the input size. On the other hand, stating scale_factor=4 will make sure the output of each spatial dimension will be 4 times the input, e.g. (30, 40) -> (120, 160).
You want to mention you need to get an output of size 240 * 320. Therefore, the spatial dimensions of your output from the decoder will be (15*4*4, 20*4*4) which conveniently is (240, 320).


Instruction: How do I prevent a lack of VRAM halfway through training a Huggingface Transformers (Pegasus) model?

Input:
I'm taking a pre-trained pegasus model through Huggingface transformers, (specifically, google/pegasus-cnn_dailymail, and I'm using Huggingface transformers through Pytorch) and I want to finetune it on my own data. This is however quite a large dataset and I've run into the problem of running out of VRAM halfway through training, which because of the size of the dataset can be a few days after training even started, which makes a trial-and-error approach very inefficient.
I'm wondering how I can make sure ahead of time that it doesn't run out of memory. I would think that the memory usage of the model is in some way proportional to the size of the input, so I've passed truncation=True, padding=True, max_length=1024 to my tokenizer, which if my understanding is correct should make all the outputs of the tokenizer of the same size per line. Considering that the batch size is also a constant, I would think that the amount of VRAM in use should be stable. So I should just be able to cut up the dataset into managable parts, just looking at the ram/vram use of the first run, and infer that it will run smoothly from start to finish.
However, the opposite seems to be true. I've been observing the amount of VRAM used at any time and it can vary wildly, from ~12GB at one time to suddenly requiring more than 24GB and crashing (because I don't have more than 24GB).
So, how do I make sure that the amount of vram in use will stay within reasonable bounds for the full duration of the training process, and avoid it crashing due to a lack of vram when I'm already days into the training process?

Output: padding=True actually doesn't pad to max_length, but to the longest sample in the list you pass to the tokenizer. To pad to max_length you need to set padding='max_length'.


Instruction: Initialise pytorch convolution layer with my own values

Input:
I want to know if there's a way to initialize pytorch convolution filter with my own values.
eg, I have a tuple [-0.8423,  0.3778][-3.1070, -2.6518] and I want to initialize a 2X2 filter with these values, how do I do that? I looked up some answers but they were mostly using torch normal distribution and others. Nothing that I can't specify.
Let me know.

Output:
You can directly assign values to weigts:
conv = nn.Conv2d(1, 1, kernel_size=2)
with torch.no_grad():
  conv.weight.data = torch.tensor([[-0.8423,  0.3778],[-3.1070, -2.6518]])  # you might need to play a bit with the dimensionality of this tensor


Instruction: How to use a custom Rank to GPU mapping in Pytorch Distributed

Input:
I'm running PyTorch distributed on a single 4-gpu node, where I am not the only one running projects. As a result, sometimes certain GPUs are too busy, while others are idle.
Let's say GPU 0 and GPU 2 are busy, but GPU 1 and GPU 3 are not, so I want to use just these GPUs. It seems that by default, Rank 0 is always GPU 0 in pytorch distributed.
How can I specify to PyTorch that Rank 0 should correspond to GPU 1, and Rank 1 should correspond to GPU 3?

Output:
You can tell any command which GPUs should be visible to them by setting the variable CUDA_VISIBLE_DEVICES. E.g.
CUDA_VISIBLE_DEVICES=2,3 python run.py <args>

This will only make GPU 2 and 3 visible to that process and thus DDP will only run on those GPUs that you specified. (Note that this is zero-indexed, i.e. 0 is the first GPU.)


Instruction: How to handle "Memory quota exceeded" in heroku app developed using streamlit and PyTorch?

Input:
I have been developing a style transfer image application using streamlit and PyTorch. The app is facing memory quota exceeded whenever high resolution image is stylized(using the app). How do I reduce the app size(its around 200+ mb)?
App code : https://github.com/pr2tik1/deep-stylize-image
App deployment : https://deep-style-images.herokuapp.com

Output:
It is not about slug size so it is nothing to do with your application size, it is about RAM you are using. Since you are using free dyno you can not process high resolution images.
You will require minimum performance dyno in my opinion since image computation require pretty much memory.


Instruction: Calling the forward method in PyTorch vs. calling the model instance

Input:
A lot of the PyTorch tutorials I've been viewing do something like this.
Define model:
class Network(nn.Module):
    def __init__():
        super().__init__()
        self.conv1 = ..
        ...

    def forward(x)
        ...
    ...

Once the Network has been instantiated (net = Network()), the people in the tutorials write net(input_data) instead of net.forward(input_data).
I tried net.forward() and it gives the same results as net().
Why is this a common practice, and also why does this work?

Output:
You should avoid calling Module.forward.
The difference is that all the hooks are dispatched in the __call__ function see this, so if you call .forward and have hooks in your model, the hooks won’t have any effect.
Inshort when you call Module.forward, pytorch hooks wont have any effect
Detailed answer can be found in this  post


Instruction: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle)` with GPU only

Input:
I'm working on the CNN with one-dimensional signal. It works totally fine with CPU device. However, when I training model in GPU, CUDA error occurred. I set os.environ['CUDA_LAUNCH_BLOCKING'] = "1" command after I got RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle). With doing this, a cublasSgemm error occurred instead of cublasCreate error.
Though the nvidia document doubt the hardware problem, I can training other CNN with images without any error. Below is my code for the data loading and set data in training model.
    idx = np.arange(len(dataset))  # dataset & label shuffle in once
    np.random.shuffle(idx)

    dataset = dataset[idx]
    sdnn = np.array(sdnn)[idx.astype(int)]

    train_data, val_data = dataset[:int(0.8 * len(dataset))], dataset[int(0.8 * len(dataset)):]
    train_label, val_label = sdnn[:int(0.8 * len(sdnn))], sdnn[int(0.8 * len(sdnn)):]
    train_set = DataLoader(dataset=train_data, batch_size=opt.batch_size, num_workers=opt.workers)

    for i, data in enumerate(train_set, 0):  # data.shape = [batch_size, 3000(len(signal)), 1(channel)] tensor

        x = data.transpose(1, 2)
        label = torch.Tensor(train_label[i * opt.batch_size:i * opt.batch_size + opt.batch_size])
        x = x.to(device, non_blocking=True)
        label = label.to(device, non_blocking=True) # [batch size]
        label = label.view([len(label), 1])
        optim.zero_grad()

        # Feature of signal extract
        y_predict = model(x) # [batch size, fc3 output] # Error occurred HERE
        loss = mse(y_predict, label)

Below is the error message from this code.
File C:/Users/Me/Desktop/Me/Study/Project/Analysis/Regression/main.py", line 217, in Processing
    y_predict = model(x) # [batch size, fc3 output]
  File "C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\ME\Desktop\ME\Study\Project\Analysis\Regression\cnn.py", line 104, in forward
    x = self.fc1(x)
  File "C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\linear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Anaconda\envs\torch\lib\site-packages\torch\nn\functional.py", line 1674, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

I've tried to solve this error for weeks but can't find the solution. If you can see anything wrong here, please let me know.

Output:
With searched with the partial keywords, I finally got the similar situation.
Because of the stability, I used the CUDA 10.2 version. The reference asked to upgrade CUDA toolkit to higher - 11.2 in my case - and problem solved!
I've deal with other training processes but this one only caused error. As the CUDA error occurred with various reasons, changes the version could be counted for solution.


Instruction: Indexing using pyTorch tensors along one specific dimension with 3 dimensional tensor

Input:
I have 2 tensors:
A with shape (batch, sequence, vocab)
and B with shape (batch, sequence).
A = torch.tensor([[[ 1.,  2.,  3.],
     [ 5.,  6.,  7.]],

    [[ 9., 10., 11.],
     [13., 14., 15.]]])

B = torch.tensor([[0, 2],
    [1, 0]])

I want to get the following:
C = torch.zeros_like(B)
for i in range(B.shape[0]):
   for j in range(B.shape[1]):
      C[i,j] = A[i,j,B[i,j]]

But in a vectorized way. I tried torch.gather and other stuff but I cannot make it work.
Can anyone please help me?

Output:
>>> import torch
>>> A = torch.tensor([[[ 1.,  2.,  3.],
...      [ 5.,  6.,  7.]],
...
...     [[ 9., 10., 11.],
...      [13., 14., 15.]]])
>>> B = torch.tensor([[0, 2],
...     [1, 0]])
>>> A.shape
torch.Size([2, 2, 3])
>>> B.shape
torch.Size([2, 2])
>>> C = torch.zeros_like(B)
>>> for i in range(B.shape[0]):
...    for j in range(B.shape[1]):
...       C[i,j] = A[i,j,B[i,j]]
...
>>> C
tensor([[ 1,  7],
        [10, 13]])
>>> torch.gather(A, -1, B.unsqueeze(-1))
tensor([[[ 1.],
         [ 7.]],

        [[10.],
         [13.]]])
>>> torch.gather(A, -1, B.unsqueeze(-1)).shape
torch.Size([2, 2, 1])
>>> torch.gather(A, -1, B.unsqueeze(-1)).squeeze(-1)
tensor([[ 1.,  7.],
        [10., 13.]])

Hi, you can use torch.gather(A, -1, B.unsqueeze(-1)).squeeze(-1).
the first -1 between A and B.unsqueeze(-1) is indicating the dimension along which you want to pick the element.
the second -1 in B.unsqueeze(-1) is to add one dim to B to make the two tensor the same dims otherwise you get RuntimeError: Index tensor must have the same number of dimensions as input tensor.
the last -1 is to reshape the result from torch.Size([2, 2, 1]) to torch.Size([2, 2])


Instruction: How to index a tensor in a for loop?

Input:
I am getting an error that says my tensor has to be a long, byte, or bool tensor. Debugging showed me that it is not getting an index [i], but the actual row [i] of the tensor x. I need that index to grab another row of another tensor y, tho.
Any ideas how to solve this?
number_batches_in = int(len(dataset_in)/batch_size)
number_batches_out = int(len(dataset_out)/batch_size)

x = torch.empty(size=(number_batches_in, 800))
y = torch.empty(size=(number_batches_out,1200), dtype=torch.float64)

for index, (x1, x2) in enumerate(dataloader_in):
    batch = torch.cat((x1, x2), 0)
    x[index] = batch

for index, (y1, y2, y3) in enumerate(dataloader_out):
    batch = torch.cat((y1, y2, y3), 0)
    y[index] = batch

model = Network(800,1200,3,800,200)
SAVE_PATH = "trained/model.dat"
epochs = 5
learning_rate = 0.001
optimizer = optim.Adam(model.parameters(),lr=learning_rate, eps=1e-08)
hist_error = []
hist_loss = []
beta = 0.5

for epoch in range(epochs):
    epoch_error = []
    epoch_loss = []
    for i in x:
        optimizer.zero_grad()
        pred = model.forward(i)
        y_true = y[i]
        loss = torch.mean(torch.sum((pred - y_true)) ** 2)
        loss.backward()
        optimizer.step()
        error = torch.mean(torch.sqrt((pred - y[i]) ** 2)).detach().numpy()
        epoch_error.append(error)
        epoch_loss.append(loss.data.detach().numpy())
    hist_error.append(np.mean(epoch_error))
    hist_loss.append(np.mean(epoch_loss))
    print("Epoch %d -- loss %f, RMS error %f " % (epoch+1, hist_loss[-1], hist_error[-1]))
torch.save(model.state_dict(), SAVE_PATH)
print("Model saved to %s" % SAVE_PATH)

Error message:
Traceback (most recent call last):
  File "/home/samim/miniconda3/envs/deep/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-54864ad18480>", line 1, in <module>
    runfile('/home/samim/Documents/train.py', wdir='/home/samim/Documents/')
  File "/home/samim/.local/share/JetBrains/PyCharm2020.3/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/samim/.local/share/JetBrains/PyCharm2020.3/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/samim/Documents/train.py", line 61, in <module>
    y_true = y[i]
IndexError: tensors used as indices must be long, byte or bool tensors

Output:
Either enumerate:
...
for j, row_x in enumerate(x):
    optimizer.zero_grad()
    pred = model.forward(row_x)
    y_true = y[j]
    loss = torch.mean(torch.sum((pred - y_true)) ** 2)
...

or better yet, zip:
...
for row_x, y_true in zip(x, y):
    optimizer.zero_grad()
    pred = model.forward(row_x)
    loss = torch.mean(torch.sum((pred - y_true)) ** 2)
...


Instruction: What is the difference between src and tgt in nn.Transformer for PyTorch?

Input:
From the docs it says to create a transformer model like this:
transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12)
src = torch.rand((10, 32, 512))
tgt = torch.rand((20, 32, 512)) # What is tgt??
out = transformer_model(src, tgt)

What is tgt mean't to be? Should tgt be the same as the src?

Output:
The transformer structure is of two components, the encoder and the decoder. The src is the input to encoder and the tgt is the input to decoder.
For example doing a machine translation task that translates English sentence to French, the src is english sequence ids and tgt is french sequence ids.


Instruction: Getting negative (inverted) image in Pytorch

Input: I want to get the negative of an image straight from the data loader and feed it as a tensor. Are there any libraries that I can use? I have tried torch transforms and didn't find any.

Output: As of 0.9, you can use torchvision.transforms.functional.invert() from the torchvision transforms library.


Instruction: Difference between Tensorflow's tf.keras.layers.Dense and PyTorch's torch.nn.Linear?

Input:
I have a quick (and possibly silly) question about how Tensorflow defines its Linear layer. Within PyTorch, a Linear (or Dense) layer is defined as, y = x A^T + b where A and b are the weight matrix and bias vector for a Linear layer (see here).
However, I can't precisely find an equivalent equation for Tensorflow! Is it the same as PyTorch or is it just y = x A + b ?
Thank you in advance!

Output:
tf.keras.layers.Dense is defined here in the tensorflow source code:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/layers/core.py#L1081
If you follow the references in its call function, it leads you to the definition of the operation used here, which is indeed a matrix multiplication of the inputs and weights plus a bias vector as expected:
https://github.com/tensorflow/tensorflow/blob/a68c6117a1a53431e739752bd2ab8654dbe2534a/tensorflow/python/keras/layers/ops/core.py#L74
outputs = gen_math_ops.MatMul(a=inputs, b=kernel)
...
outputs = nn_ops.bias_add(outputs, bias)


Instruction: Bubble up non null rows of a 3-D Tensor PyTorch

Input:
I am having some problem trying to convert the following function into a function only manipulating tensors.
def valid_sequence_output(sequence_output, valid_mask):
    bs, max_len, feat_dim = sequence_output.shape
    valid_output = torch.zeros(bs, max_len, feat_dim, dtype=torch.float32)
    for i in range(bs):
        jj = -1
        for j in range(max_len):
            if valid_mask[i][j].item() == 1:
                jj += 1
                valid_output[i][jj] = sequence_output[i][j]
    return valid_output

where the input tensors can be created as follow:
size = ((2,5,2))
sequence_output = torch.randint(0, 250, size=size)
valid_mask = torch.randint(0, 2, size=size[:2])

I basically aim at “bubbling up” the non null rows of sequence_output. As an example having sequence_output equal to:
tensor([[[  0,   0],
         [ 15,  47],
         [124, 230],
         [ 0,  0],
         [  65,   31]],
        [[  0,   0],
         [  0,   0],
         [  0,   0],
         [139, 228],
         [224, 205]]])

I am trying to obtain the following tensor:
tensor([[[ 15,  47],
         [124, 230],
         [ 65,  31],
         [  0,   0],
         [  0,   0]],
        [[139, 228],
         [224, 205],
         [  0,   0],
         [  0,   0],
         [  0,   0]]])

If somebody has a suggestion on how to do this, I would really appreciate :D

Output:
I managed to come up with a quite nasty solution (which he's probably suboptimal) by constructing a matrix A of zeros and ones that will swap rows of X when performing a matrix multiplication...
def vso(seq_out, valid_mask):
    X = torch.where(valid_mask.unsqueeze(-1) == 1, seq_out, torch.zeros_like(seq_out))
    bs, max_len, _ = X.shape
    tu = torch.unique(torch.nonzero(X)[:, :2], dim=0)
    batch_axis = tu[:, 0]
    rows_axis = tu[:, 1]
    a = torch.arange(bs).repeat(batch_axis.shape).reshape(batch_axis.shape[0], -1).T
    T = torch.cumsum(batch_axis == a, dim=1) - 1
    cols_axis = T[batch_axis, torch.arange(batch_axis.shape[0])]
    A = torch.zeros((bs, max_len, max_len))
    A[(batch_axis, cols_axis, rows_axis)] = 1
    valid_output = torch.matmul(A, X)
    return valid_output

Still looking for better answers though !


Instruction: TorchScript requires source access in order to carry out compilation for collections.deque

Input:
I'm trying to convert PyTorch FOMM model to TorchScript. As soon as I started to annotate some classes with @torch.jit.script I've got an error:
OSError: Can't get source for <class 'collections.deque'>. TorchScript requires source access in order to carry out compilation, make sure original .py files are available.
As I understand that class implemented in CPython therefore cannot be read by TorchScript compiler. I failed to find any pure-Python implementation. How can I overcome this issue?
Here is the class I'm trying to annotate:
import queue
import collections
import threading
import torch

@torch.jit.script
class SyncMaster(object):
    """An abstract `SyncMaster` object.

    - During the replication, as the data parallel will trigger an callback of each module, all slave devices should
    call `register(id)` and obtain an `SlavePipe` to communicate with the master.
    - During the forward pass, master device invokes `run_master`, all messages from slave devices will be collected,
    and passed to a registered callback.
    - After receiving the messages, the master device should gather the information and determine to message passed
    back to each slave devices.
    """

    def __init__(self, master_callback):
        """

        Args:
            master_callback: a callback to be invoked after having collected messages from slave devices.
        """
        self._master_callback = master_callback
        self._queue = queue.Queue()
        self._registry = collections.OrderedDict()
        self._activated = False

    def __getstate__(self):
        return {'master_callback': self._master_callback}

    def __setstate__(self, state):
        self.__init__(state['master_callback'])

    def register_slave(self, identifier):
        """
        Register an slave device.

        Args:
            identifier: an identifier, usually is the device id.

        Returns: a `SlavePipe` object which can be used to communicate with the master device.

        """
        if self._activated:
            assert self._queue.empty(), 'Queue is not clean before next initialization.'
            self._activated = False
            self._registry.clear()
        future = FutureResult()
        self._registry[identifier] = _MasterRegistry(future)
        return SlavePipe(identifier, self._queue, future)

    def run_master(self, master_msg):
        """
        Main entry for the master device in each forward pass.
        The messages were first collected from each devices (including the master device), and then
        an callback will be invoked to compute the message to be sent back to each devices
        (including the master device).

        Args:
            master_msg: the message that the master want to send to itself. This will be placed as the first
            message when calling `master_callback`. For detailed usage, see `_SynchronizedBatchNorm` for an example.

        Returns: the message to be sent back to the master device.

        """
        self._activated = True

        intermediates = [(0, master_msg)]
        for i in range(self.nr_slaves):
            intermediates.append(self._queue.get())

        results = self._master_callback(intermediates)
        assert results[0][0] == 0, 'The first result should belongs to the master.'

        for i, res in results:
            if i == 0:
                continue
            self._registry[i].result.put(res)

        for i in range(self.nr_slaves):
            assert self._queue.get() is True

        return results[0][1]

    @property
    def nr_slaves(self):
        return len(self._registry)

Output: Switched TorchScript generation method from torch.jit.script to torch.jit.trace and it worked, no need in annotating anything. Alternatively torch.onnx.export works sometimes.


Instruction: RuntimeError('dot : expected both vectors to have same dtype, but found Double and Float

Input:
I am trying to perform this operation:
L_c += 1 - torch.abs(torch.dot(nuNormalized, Nu))

where:

nuNormalized -> tensor([ 0.3790, -0.4208,  0.8242], dtype=torch.float64)

Nu -> tensor([-0.9961, -0.9961, -0.9961], device='cuda:0')

I am getting this error:

(<class 'RuntimeError'>, RuntimeError('dot : expected both vectors to have same dtype, but found Double and Float',), <traceback object at 0x7f4d276155c8>)

Any suggestions please? Thanks!

Output:
cast nuNormalized to float from double.
also you might consider moving both to the same device.


Instruction: How to find the best value for mean and STD of Normalize in torchvision.transforms

Input:
I have started working with PyTorch and cannot figure it how I am supposed to find mean and std as the input parameters of normalise.
I have seen this
transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)) #https://pytorch.org/vision/stable/transforms.html#

and in another example:
transformation = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5,0.5,0.5],std=[0.5,0.5,0.5])
])  #https://github.com/MicrosoftDocs/ml-basics/blob/master/challenges/05%20%20-%20Safari%20CNN%20Solution%20(PyTorch).ipynb

so, how am I supposed to know or get these values if I have a set of images?
Are these three parameters are related to R G B also?

Output:
suppose you have already X_train which is a list of numpy matrix eg. 32x32x3:
X_train = X_train / 255 #normalization of pixels
train_mean = X_train.reshape(3,-1).mean(axis=1)
train_std = X_train.reshape(3,-1).std(axis=1)

then you can pass this last 2 variables in your normalizer transformer :
transforms.Normalize(mean = train_mean ,std = train_std)


Instruction: Torch select index for mult dimensional vector

Input:
Suppose I have the following data:
import torch
torch.manual_seed(42)
logits = torch.randn(2, 5, 3)
idx = torch.randint(0, 3, (2, 5))

What I want to do is the following:
[[logits[i,j,idx[i][j]] for j in range(len(idx[i]))] for i in range(len(idx))]

However, this is clearly inefficient.
The closest that I came to this was doing this, but this again looks pretty ugly:
new_idx = torch.stack([idx]*logits.shape[-1], dim=-1)
logits.gather(dim=-1, index=new_idx)

where in the above case the desired output is replicated 3 times.
In terms of practical use case, I'm thinking of a language model where the logits are of shape (batch_size, sequence_len, vocabulary), and the indices are simply (batch_size, sequence_len). Apologies if this has been asked before but I couldn't find anything.

Output:
same with this answer
torch.gather(logits, -1, idx.unsqueeze(-1))
>>> [[logits[i,j,idx[i][j]] for j in range(len(idx[i]))] for i in range(len(idx))]
[[tensor(0.9007), tensor(0.6784), tensor(-0.0431), tensor(-1.4036), tensor(-0.7279)], [tensor(-0.2168), tensor(1.7174), tensor(-0.4245), tensor(0.9956), tensor(-1.2742)]]

>>> torch.gather(logits, -1, idx.unsqueeze(-1))
tensor([[[ 0.9007],
         [ 0.6784],
         [-0.0431],
         [-1.4036],
         [-0.7279]],

        [[-0.2168],
         [ 1.7174],
         [-0.4245],
         [ 0.9956],
         [-1.2742]]])


Instruction: Sequence to Sequence Loss

Input:
I'm trying to figure out how sequence to sequence loss is calculated. I am using the huggingface transformers library in this case, but this might actually be relevant to other DL libraries.
So to get the required data we can do:
from transformers import EncoderDecoderModel, BertTokenizer
import torch
import torch.nn.functional as F
torch.manual_seed(42)

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
MAX_LEN = 128
tokenize = lambda x: tokenizer(x, max_length=MAX_LEN, truncation=True, padding=True, return_tensors="pt")

model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints
input_seq = ["Hello, my dog is cute", "my cat cute"]
output_seq = ["Yes it is", "ok"]
input_tokens = tokenize(input_seq)
output_tokens = tokenize(output_seq)

outputs = model(
    input_ids=input_tokens["input_ids"],
    attention_mask=input_tokens["attention_mask"],
    decoder_input_ids=output_tokens["input_ids"],
    decoder_attention_mask=output_tokens["attention_mask"],
    labels=output_tokens["input_ids"],
    return_dict=True)

idx = output_tokens["input_ids"]
logits = F.log_softmax(outputs["logits"], dim=-1)
mask = output_tokens["attention_mask"]

Edit 1
Thanks to @cronoik I was able to replicate the loss calculated by huggingface as being:
output_logits = logits[:,:-1,:]
output_mask = mask[:,:-1]
label_tokens = output_tokens["input_ids"][:, 1:].unsqueeze(-1)
select_logits = torch.gather(output_logits, -1, label_tokens).squeeze()
huggingface_loss = -select_logits.mean()

However, since the last two tokens of the second input is just padding, shouldn't we calculate the loss to be:
seq_loss = (select_logits * output_mask).sum(dim=-1, keepdims=True) / output_mask.sum(dim=-1, keepdims=True)
seq_loss = -seq_loss.mean()

^This takes into account the length of the sequence of each row of outputs, and the padding by masking it out. Think this is especially useful when we have batches of varying length outputs.

Output:
ok I found out where I was making the mistakes. This is all thanks to this thread in the HuggingFace forum.

The output labels need to have -100 for the masked version. The transoformers library does not do it for you.
One silly mistake I made was with the mask. It should have been output_mask = mask[:, 1:] instead of :-1.

1. Using Model
We need to set the masks of output to -100. It is important to use clone as shown below:
labels = output_tokens["input_ids"].clone()
labels[output_tokens["attention_mask"]==0] = -100

outputs = model(
    input_ids=input_tokens["input_ids"],
    attention_mask=input_tokens["attention_mask"],
    decoder_input_ids=output_tokens["input_ids"],
    decoder_attention_mask=output_tokens["attention_mask"],
    labels=labels,
    return_dict=True)

2. Calculating Loss
So the final way to replicate it is as follows:
idx = output_tokens["input_ids"]
logits = F.log_softmax(outputs["logits"], dim=-1)
mask = output_tokens["attention_mask"]

# shift things
output_logits = logits[:,:-1,:]
label_tokens = idx[:, 1:].unsqueeze(-1)
output_mask = mask[:,1:]

# gather the logits and mask
select_logits = torch.gather(output_logits, -1, label_tokens).squeeze()
-select_logits[output_mask==1].mean(), outputs["loss"]

The above however ignores the fact that this comes from two different lines. So an alternate way of calculating loss could be:
seq_loss = (select_logits * output_mask).sum(dim=-1, keepdims=True) / output_mask.sum(dim=-1, keepdims=True)
seq_loss.mean()


Instruction: (Differentiable Image Sampling) Custom Integer Sampling Kernel, Spatial Transformer Network

Input:
I was looking through the Spatial Transformer Network paper, and I am trying to implement a custom grid_sample function (inheriting the autograd.Function class) in PyTorch for the Integer Sampling Kernel.
While defining the backward function, I have come across the following conundrum.
Given that the integer sampling works as the following:

I think that the gradients w.r.t the input map and the transformed grid (x_i^s, y_i^s) should be like the following:
Gradient w.r.t. input map:

Gradient w.r.t transformed grid (x_i^s):

Gradient w.r.t transformed grid (y_i^s):

as the derivative of the Kronecker delta function is zero (I'm unsure about this!! - HELP)
Derivative of the Kronecker delta?

Thus I am reaching a conclusion that the gradient w.r.t to the input should be: a tensor of the same size as the input filled with ones if the pixel was sampled and 0 if it wasn't sampled, and the gradient w.r.t the transformed grid should be a tensor full of zeros.
However, if the gradient of the transformed grid is 0, then due to the chain rule, no information will be passed on to the layers before the integer sampler. Therefore I think the derivate with respect to the grid should be something else. Could anybody point out what I'm doing wrong?
Many thanks in advance!

Output:
For future reference, and for those who might have had similar questions to the one I posted.

I've emailed Dr Jaderberg (one of the authors of the 'Spatial Transformer Networks') about this question and he has confirmed: "that the gradient wrt the coordinates for integer sampling is 0.". So I wasn't doing anything wrong, and it was right all along!
He was very kind in his response and expressed that integer sampling was mentioned in the paper to introduce the bilinear sampling scheme, and have given insights into how to possibly implement integer sampling if I really wanted to:
"you could think about using some numerical differentiation techniques (e.g. look at difference of x to its neighbours). This would assume smoothness in the image wrt coordinates."
So with great thanks to Dr Jaderberg, I'm happy to close this question.
I guess thinking about how I'd use numerical methods to implement the integer kernel for the sampling function is another challenge for myself, but until then I guess the bilinear sampler is my friend! :)


Instruction: Why are PyTorch "convolutions" implemented as cross-correlations?

Input:
PyTorch convolutions are actually implemented as cross-correlations. This shouldn't produce issues in training a convolution layer, since one is just a flipped version of the other (and hence the learned function will be equally powerful), but it does prove an issue when:

trying to implement an actual convolution with the functional library
trying to copy the weights of an actual convolution from another deep learning library

The authors say the following in Deep Learning with PyTorch:

Convolution, or more precisely, discrete convolution1...
1. There is a subtle difference between PyTorch's convolution and mathematics' convolution: one argument's sign is flipped. If we were in a pedantic mood, we could call PyTorch's convolutions discrete cross-correlations.

But they don't explain why it was implemented like this. Is there a reason?
Maybe something similar to how the PyTorch implementation of CrossEntropyLoss isn't actually cross entropy but an analogous function taking "logits" as inputs instead of raw probabilities (to avoid numerical instability)?

Output:
I think the reason is simpler. As you said, convolution is the flipped version of cross-correlation, but that's not a problem in the context of training a CNN. So we can just avoid doing the flipping, which simplifies the code and reduces the computation time:

The advantage of cross-correlation is that it avoids the additional step of flipping the filters to perform the convolutions.

Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs

Flipping the kernel won't have any effect on the mathematical stability. The operations remain the same.


Instruction: How to define a “don't care” class in Pytorch?

Input:
I have a time series classification task in which I should output a classification of 3 classes for every time stamp t.
All data is labeled per frame.
In the data set are more than 3 classes [which are also imbalanced].
My net should see all samples sequentially, because it uses that for historical information.
Thus, I can't just eliminate all irrelevant class samples at preprocessing time.
In case of a prediction on a frame which is labeled differently than those 3 classes, I don't care about the result.

How to do this correctly in Pytorch?

Output:
Following from this discussion, which was not google searchable, there are two options, both are options of the CrossEntropyLoss:
Option 1
If there is only one class to ignore, use ignore_index=class_index when instantiating the loss.
Option 2
If there are more classes, use weight=weights, with weights.shape==n_classes and torch.sum(weights[ignored_classes]) == 0


Instruction: Strange Cuda out of Memory behavior in Pytorch

Input:
Edit: SOLVED- Problem relied on the number of workers, lowered them, problem solved
I am using a 24GB Titan RTX and I am using it for an image segmentation Unet with Pytorch,
it is always throwing Cuda out of Memory at different batch sizes, plus I have more free memory than it states that I need, and by lowering batch sizes, it INCREASES the memory it tries to allocate which doesn't make any sense.
here is what I tried:
Image size = 448, batch size = 8

"RuntimeError: CUDA error: out of memory"

Image size = 448, batch size = 6

"RuntimeError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 0; 24.00 GiB total capacity; 2.06 GiB already allocated; 19.66 GiB free; 2.31 GiB reserved in total by PyTorch)"

is says it tried to allocate 3.12GB and I have 19GB free and it throws an error??
Image size = 224, batch size = 8

"RuntimeError: CUDA out of memory. Tried to allocate 28.00 MiB (GPU 0; 24.00 GiB total capacity; 2.78 GiB already allocated; 19.15 GiB free; 2.82 GiB reserved in total by PyTorch)"

Image size = 224, batch size = 6

"RuntimeError: CUDA out of memory. Tried to allocate 344.00 MiB (GPU 0; 24.00 GiB total capacity; 2.30 GiB already allocated; 19.38 GiB free; 2.59 GiB reserved in total by PyTorch)"

reduced batch size but tried to allocate more ???
Image size = 224, batch size = 4

"RuntimeError: CUDA out of memory. Tried to allocate 482.00 MiB (GPU 0; 24.00 GiB total capacity; 2.21 GiB already allocated; 19.48 GiB free; 2.50 GiB reserved in total by PyTorch)"

Image size = 224, batch size = 2

"RuntimeError: CUDA out of memory. Tried to allocate 1.12 GiB (GPU 0; 24.00 GiB total capacity; 1.44 GiB already allocated; 19.88 GiB free; 2.10 GiB reserved in total by PyTorch)"

Image size = 224, batch size = 1

"RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 24.00 GiB total capacity; 894.36 MiB already allocated; 20.94 GiB free; 1.03 GiB reserved in total by PyTorch)"

Even with stupidly low image sizes and batch sizes...

Output: SOLVED- Problem relied on the number of workers, lowered them, problem solved


Instruction: Pytorch: test loss becoming nan after some iteration

Input: I am trying to train a deep learning architecture, the model trains perfectly. I am testing after each epoch. For 7 epoch all the loss and accuracy seems okay but at 8 epoch during the testing test loss becomes nan. I have checked my data, it got no nan. Also my test accuracy is higher than train which is weird. Train data size is 37646 and test is 18932 so it should be enough. Before becoming nan test started to become very high around 1.6513713663602217e+30. This is really weird and I don't understand why is happening. Any help or suggestion is much appreciated.

Output:
Assuming that a very high learning rate isn't the cause of the problem, you can clip your gradients before the update, using PyTorch's gradient clipping.
Example:
optimizer.zero_grad()
loss, hidden = model(data, hidden, targets)
loss.backward()

torch.nn.utils.clip_grad_norm_(model.parameters(), clip_value)
optimizer.step()

This is the first thing to do when you have a NaN loss, if of course you have made sure than you don't have NaNs elsewhere, e.g. in your input features. I have made use of gradient clipping in cases where increasing the learning rate caused NaNs, but still wanted to test a higher learning rate. Decreasing the learning rate could also solve your problem, but I'm guessing that you have already tried this.
Empirically, I set clip_value = 5 most of the times, and then see its (usually non-significant) impact on performance. Feel free to experiment with different values.


Instruction: Pixel operations in batches

Input:
I have a batch of depth images, shape -> [B, 1, H, W]. For each pixel in each image of the batch I need to perform:
X = d * Kinverse @ [u, v, 1]   #therefore X is in R^3
where d is float tensor[0;1] representing depth at pixel u,v; Kinverse is a constant 3X3 matrix and u, v refer to the pixel column and row respectively.
Is there some way I can vectorize the operation to obtain X(u+1,v), X(u,v) and X(u,v+1) for all the images in the batch.
I eventually need to take this cross product:
{X(u+1,v) - X(u,v)} x {X(u, v+1) - X(u,v)}
Thanks for the help!

Output:
You can use torch.meshgrid to produce the u and v tensors. Once you have them, you can use torch.einsum to do the batched matrix multiplication with Kinverse.
Finally, you can use torch.cross to compute the cross product:
u, v = torch.meshgrid(*[torch.arange(s_, dtype=d.dtype, device=d.device) for s_ in d.shape[2:]])
# make a single 1x1xHxW for [u v 1] per pixel:
uv = torch.cat((u[None, None, ...], v[None, None, ...], torch.ones_like(u)[None, None, ...]), dim=1)
# compute X
X = d * torch.einsum('ij,bjhw->bihw',Kinverse,uv)
# the cross product
out = torch.cross(X[..., 1:, :-1] - X[..., :-1, :-1], X[..., :-1, 1:] - X[..., :-1, :-1], dim=1)


Instruction: PyTorch: saving both weights and model definition

Input:
During prototyping, I often perform numerous changes to a PyTorch model. For instance, suppose the first model I am experimenting with is:
class Model(nn.Module):
    def __init__(self, **args):
        super().__init__()
        self.l1 = nn.Linear(128, 1)

then I will add another layer:
class Model(nn.Module):
    def __init__(self, **args):
        super().__init__()
        self.l1 = nn.Linear(128, 32)
        self.l2 = nn.Linear(32, 1)

or maybe add some convolutions, and so on.
The problem is that I often get disorganized the more experiments I perform, as I haven't found a straightforward way of saving both model definition and its weights so that I can load a previous state.
I know I can do:
torch.save({'model': Model(), 'state': model.state_dict()}, path)
# or directly
torch.save(model, path)

but then loading the model also requires that the model class (here, Model) to exist in the current file.
In Keras you can simply do:
model = ...  # Get model (Sequential, Functional Model, or Model subclass)
model.save('path/to/location')

which saves model's architecture/config and weights, among other things. This means that you can load the model without having defined the architecture:
model = keras.models.load_model('path/to/location')

Refering to Keras model saving:

The SavedModel and HDF5 file contains:

the model's configuration (topology)
the model's weights
the model's optimizer's state (if any)

Thus models can be reinstantiated in the exact same state, without any
of the code used for model definition or training.

This is what I want to achieve in PyTorch.
Is there a similar approach for PyTorch? What is the best practice for these situations?

Output:
As Pytorch provides a huge amount of flexibility in the model, it will be challenging to save the architecture along with the weights in a single file. Keras models are usually built solely by stacking keras components, but pytorch models are orchestrated by the library consumer in their own way and therefore can contain any sort of logic.
I think you have three choices:

Come up with a organised schema for your experiments so that losing the model definition is less likely. You could go for something as simple as a file named through a schema that only defines each model. I would recommend this approach as this level of organisation would likely benefit your prototyping in other ways and the overhead is minimal.

Try and save the code along with the pickle file. Although potentially possible, I think this would lead you down a rabbit-hole with a lot of potential problems.

Use a different standardised way of saving the model, such as onnx. I would recommend this route if you do not want to go with option 1. Onnx does allow you to save a pytorch model's architecture along with its weights but comes with a few drawbacks. For example, it only supports some operations so completely custom forward methods or use of non-matrix operations may not work.


Instruction: Longformer get last_hidden_state

Input:
I am trying to follow this example in the huggingface documentation here https://huggingface.co/transformers/model_doc/longformer.html:
import torch
from transformers import LongformerModel, LongformerTokenizer
model = LongformerModel.from_pretrained('allenai/longformer-base-4096')
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
SAMPLE_TEXT = ' '.join(['Hello world! '] * 1000)  # long input document
input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1
# Attention mask values -- 0: no attention, 1: local attention, 2: global attention
attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
global_attention_mask = torch.zeros(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to global attention to be deactivated for all tokens
global_attention_mask[:, [1, 4, 21,]] = 1  # Set global attention to random tokens for the sake of this example
                                    # Usually, set global attention based on the task. For example,
                                    # classification: the <s> token
                                    # QA: question tokens
                                    # LM: potentially on the beginning of sentences and paragraphs
outputs = model(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask, output_hidden_states= True)
sequence_output = outputs[0].last_hidden_state
pooled_output = outputs.pooler_output

I suppose that this would return a document embedding for the sample text.
However, I run into the following error:
AttributeError: 'Tensor' object has no attribute 'last_hidden_state'

Why isnt it possible to call last_hidden_state?

Output:
Do not select via index:
sequence_output = outputs.last_hidden_state

outputs is a LongformerBaseModelOutputWithPooling object with the following properties:
print(outputs.keys())

Output:
odict_keys(['last_hidden_state', 'pooler_output', 'hidden_states'])

Calling outputs[0] or outputs.last_hidden_state will both give you the same tensor, but this tensor does not have a property called last_hidden_state.


Instruction: Is there a way to only display frames from a video that only have detected object bounding boxes

Input:
I'm using a video for testing my custom yolov5s object detection model on test data. However, not every frame in the video has a valid object to be detected.
[ When the inference testing is ran on the video, the image below shows when the model finds a frame that has an object that can be detected and classified
As you can see, frame 4401 has detected a yield traffic sign but frame 4399 and 4400 haven't detected anything. My question is how do I process through 4500 frames of a video and only display frames that have detected objects. I have extracted the frames as JPEGS but only want the valid JPEGS to evaluate test performance? Better still, is it possible to extract the data from relevant frames to a text file?
If anyone could help me out this would be a great help and time saver, many thanks in advance.
the new error

Output:
from shutil import move
import os

txt_dir = "txtfiles"
images_folder = "images_folder"
move_des = "move_imgs_path"

for file in os.listdir(txt_dir):
    if file.endswith('txt'):
          frame_name = file.split('.')[0]+ ".jpeg"
          print(frame_name)
          move(os.path.join(images_folder, frame_name), move_des)


Instruction: NER tagging schema for non-contiguous tokens

Input:
The most common tagging procedure for NER is IOB. But it seems that this kind of tagging is limited to cases where tokens from the same entity are contiguous.
So for instance,
Jane Smith is walking in the park would be tagged as: B-PER I-PER O O O O O
And here my PER entity is the concatenation of [Jane, Smith]
If we tweak the example:
Jane and James Smith are walking in the park
B-PER O B-PER I-PER O O O O O
Now the issue is that the entities we would get are [Jane] and [James, Smith] because the IOB tagging does not allow to link Jane to Smith.
Is there any tagging schema that would allow to mark as entities both [Jane, Smith] and [James, Smith]?

Output:
First, about doing this without a new data format:
There are a paper and repo about doing this using TextAE for this:
paper
repo
However, looking at their examples and yours, it seems like you could improve on what they did by using dependency parsing. If you look at the dependency parse of "Jane and James Smith are walking in the park", you can see that spaCy understands that Jane is conjoined with Smith. So after running entity extraction, you could do a dependency parse step, then edit your entities based on that.
Now, to answer the real question. I have seen multi-dimensional labels that work in the following way (assume you have a maximum of ten entities per sentence:
empty = [0,0,0,0,0,0,0,0,0]

tokens = ["Jane", "and", "James", "Smith", "are", "walking", "in", "the", "park"]
labels = [
    [1, 0, 0, 1, 0, 0, 0, 0, 0],
    [0, 0, 1, 1, 0, 0, 0, 0, 0],
]
labels = labels + [empty] * (10-len(labels))

If you have more than one entity type, you can use those instead of just 1.
This format works better with BERT anyway, since the BIO format is a pain when you have to split up tokens into BPE anyway.


Instruction: Understanding pos_weight argument in BCEWithLogitsLoss

Input:
I am trying to understand how the pos_weight argument is being used in BCEWithLogitsLoss in order to be able to correctly define the pos_weight Tensor. The documentation only mentions: "a weight of positive examples. Must be a vector with length equal to the number of classes.". As I was not able to get a sufficient understanding by looking at the code (the actual code is hidden behind multiple function loaders and I didn't even manage to reach a point where pos_weight is used), I have several questions regarding the pos_weight argument:

Is the weight for negative samples always 1?
If the weight for negative samples is always 1 and assuming I want every sample to contribute equally to the loss I would do something like the following. Let l = [100, 10, 5, 15] where l[0] is the amount of negative samples and l[1:] is the amount of positive samples from each label. In pseudo code this looks like:

l = [100, 10, 5, 15]
lcm = LCM(l)  # 300
weights = lcm / l  # weights = [3, 30, 60, 20]
weights = weights / l[0]  # weights = [1, 10, 20, 6.6667]
positive_weights = weights[1:]  # [10, 20, 6.66667]
criterion = nn.BCEWithLogitsLoss(pos_weight=positive_weights)

Can someone please confirm if my understanding of how pos_weight should be used is correct?

Output:
I didn't manage to find a definitive answer but from my experiments it seems I was right, i.e. negative sample weights are considered to be 1 and increasing the weights past 1 for positive samples makes them more important. Additionally, the LCM approach for figuring out the weights is actually both not good and not needed. First it is bad because if you have a lot of classes with different number of samples the LCM can be an incredibly large number so very often computing it exactly is not even feasible. Additionally, it is not needed because you achieve the same weights by simply dividing the amount of negative samples with the amount of positive samples for every label and obtain the weights in that manner. Here is a short code snippet on how you can obtain the weights if you have your data in pandas DataFrame. Obtaining the weights in this way results in a 'balanced' dataset where ideally you optimize for both precision and recall.
def get_train_dataset_pos_weights(self):
    """Returns weights for positive samples used for unbalanced datasets."""

    num_pos_samples = self.train_df[self.label_columns].sum().sum()
    num_neg_samples = len(self.train_df) - num_pos_samples
    assert len(self.train_df) == (num_neg_samples + num_pos_samples)
    num_pos_samples_per_label = np.array(
        list(self.train_df[self.label_columns].sum()))
    weights = num_neg_samples / num_pos_samples_per_label

    return torch.from_numpy(weights)


Instruction: Dot product along a dimension

Input:
I have two tensors of shape [B, 3 , 240, 320] where B represents the batch size 3 represents the channels, 240 the height(H), 320 the width(W).
I need to find the dot product along the channels dimension(3 channels) thus the resulting tensor would be of shape [B, 1 , 240, 320].
My tensors have float32 elements in gpu(cuda to backprop).
Can you all please suggest how I can do that?
Thanks!
More clarification:
Let's say we have B=10, H=100, W=200.
So from the above would be common for both the first and seconds tensors. If we keep B, H, W constant we get a 1D vector as the resultant tensor(with 3 elements). I need to take the dot product of these two vectors. Thus the resultant tensor is of dimension [B, 1, 240, 320]

Output:
Dot product is the summation of multiplication of values in two vectors:

So I am guessing you want to multiply all values along the channel dimension and need to find the summation of the result, please correct me if my understanding is wrong.
import torch

t1 = torch.rand(10, 3, 240, 320)
t2 = torch.rand(10, 3, 240, 320)

# Multiply two tensors and sum along the channel dimension
multp  = t1 * t2
summed = multp.sum(dim = 1, keepdim = True)

print(summed.shape)  # torch.Size([10, 1, 240, 320])


Instruction: Pytorch Conv1d on simple 1d signal

Input:
1d-convolution is pretty simple when it is done by hand. However, I want to implement what is done here using nn.Conv1d and it is not simple for me to do it. In this example h=[1,2,-1], x=[4,1,2,5] and the output is going to be y=[4,9,0,8,8,-5]. To do it using Pytorch we need to define h=nn.Conv1d(in, out, k) and x=torch.tensor(*) and y=h(x) should be the result.
Note: please do not use nn.Conv2d to implement it.

Output:
First, you should be aware that the term "convolution" used in basically all literature related to convolutional neural networks (CNNs) actually corresponds to the correlation operation not the convolution operation.
The only difference (for real-valued inputs) between correlation and convolution is that in convolution the kernel is flipped/mirrored before sliding it across the signal, whereas in correlation no such flipping occurs.
There are also some extra operations that convolution layers in CNNs perform that are not part of the definition of convolution. They apply an offset (a.k.a. bias), they operate on mini-batches, and they map multi-channel inputs to multi-channel outputs.
Therefore, in order to recreate a convolution operation using a convolution layer we should (i) disable bias, (ii) flip the kernel, and (iii) set batch-size, input channels, and output channels to one.
For example, a PyTorch implementation of the convolution operation using nn.Conv1d looks like this:
import torch
from torch import nn

x = torch.tensor([4, 1, 2, 5], dtype=torch.float)
k = torch.tensor([1, 2, -1], dtype=torch.float)

# Define these constants to differentiate the various usages of "1".
BATCH_SIZE, IN_CH, OUT_CH = 1, 1, 1

# Pad with len(k)-1 zeros to ensure all non-zero outputs are computed.
h = nn.Conv1d(IN_CH, OUT_CH, kernel_size=len(k), padding=len(k) - 1, bias=False)

# Copy flipped k into h.weight.
# h.weight is shape (OUT_CH, IN_CH, kernel_size), reshape k accordingly.
# Perform copy inside no_grad context to avoid autograd issues.
with torch.no_grad():
    h.weight.copy_(torch.flip(k, dims=[0]).reshape(OUT_CH, IN_CH, -1))

# Input shape to h is assumed to be (BATCH_SIZE, IN_CH, SIGNAL_LENGTH), reshape x accordingly.
# Output shape of h is (BATCH_SIZE, OUT_CH, OUTPUT_LENGTH), reshape output to 1D signal.
y = h(x.reshape(BATCH_SIZE, IN_CH, -1)).reshape(-1)

which results in
>>> print(y)
tensor([ 4.,  9.,  0.,  8.,  8., -5.], grad_fn=<ViewBackward>)


Instruction: pytorch question about tensor shape. And how to reshape a tensor

Input: When I do  print("action shape:, ", action.shape) for my tensor action I got (64,). It is the same as (1, 64)? And how do I reshape its size to (64,1)?

Output:
Technically it is not the same shape and in pytorch you will get an error if you have things that need a shape of (64,) but you give it (1,64) but it is easy to change it to (64,) by squeezing it. To reshape it to a size of (64, 1) you can do this
action = action.unsqueeze(1)
# or
action = action.view(-1, 1)

either will work but I would recommend the first one.


Instruction: when setting .eval() my model performs worse than when I set .train()

Input:
During the training phase, I select the model parameters with the best performance metric.
if performance_metric.item()>max_performance:
        max_performance= performance_metric.item()
        torch.save(neural_net.state_dict(), PATH+'/best_model.pt')

This is the neural network model used:
class Neural_Net(nn.Module):
        def __init__(self, M,shape_input,batch_size):
            super(Neural_Net, self).__init__()
            self.lstm = nn.LSTM(shape_input,M)
            #self.dense1 = nn.Linear(shape_input,M)
            self.dense1 = nn.Linear(M,M) #Used with the LSTM
            torch.nn.init.xavier_uniform_(self.dense1.weight)
            self.dense2 = nn.Linear(M,M)
            torch.nn.init.xavier_uniform_(self.dense2.weight)
            self.dense3 = nn.Linear(M,1)
            torch.nn.init.xavier_uniform_(self.dense3.weight)
            self.drop   = nn.Dropout(0.7)
            self.bachnorm1 = nn.BatchNorm1d(M)
            self.relu = nn.ReLU()
            self.sigmoid = nn.Sigmoid()

            self.hidden_cell = (torch.zeros(1,batch_size,M),torch.zeros(1,batch_size,M))

        def forward(self, x):
            lstm_out, self.hidden_cell = self.lstm(x.view(1 ,len(x), -1), self.hidden_cell)
            x = self.drop(self.relu(self.dense1(self.bachnorm1(lstm_out.view(len(x), -1)))))
            x = self.drop(self.relu(self.dense2(x)))
            x = self.relu(self.dense3(x))
            return x

After that I load the model with the best parameters and set the evaluation mode:
neural_net.load_state_dict(torch.load(PATH+'/best_model.pt'))
neural_net.eval()

The results are completely random. When I set train() the performance is similar to the selected best model parameter.
There is an important aspect of the eval() that I am forgetting? Is the batch normalization correctly used? I am using a batch the same size as in the training phase for the test phase.

Output:
Without knowing your batch size, training/test dataset size, or the training/test dataset discrepancies, this issue has been discussed on the pytorch forums previously here.
In my experience, it sounds very much like your latent training data representation in your model is significantly different to your validation data representation. The main advice I can provide is for you to try reducing the momentum of your batchnorm layer. It might be worth substituting a layernorm layer instead (which doesn't track a running mean/standard deviation) OR setting track_running_stats=False in the batchnorm1d function and seeing if the problem persists.


Instruction: Visualizing CNN

Input:
Hi I'm trying to visualize CNN.
I've been going through https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html
to study CNN by visualizing the structure. What I couldn't understand is its dimension.

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

So the code part is supposed to be the CNN model structure from the image.
What I don't get is this.

Convolutions happen from the input to C1 and 3*3 kernel was used. In this case, shouldn't the dimension of C1 be 30 X 30 instead of 28 X 28?
According to the image, dimension of the input to layer F5 is 16 X 5 X 5 but the code says otherwise. It seems like layer F5 is taking an input of dimension 16 X 6 X 6.

I'm not sure whether I'm taking it incorrectly or the image is wrong.

Output:
I'm pretty sure the image is wrong.
If you check the documentation of Conv2d. Using the equation there, the first convolution layer should output (batch_size, 6, 30, 30). Running the model also confirms my conclusion.
The image should be modified to:
INPUT: 1 x 32 x 32
C1: 6 x 30 x 30
S2: 6 x 15 x 15
C2: 16 x 13 x 13
S2: 16 x 6 x 6


Instruction: Why does Pytorch EmbeddingBag with mode "max" not accept `per_sample_weights`?

Input:
Pytorch's EmbeddingBag allows for efficient lookup + reduce operations on varying length collections of embedding indices. There are 3 modes: "sum", "average" and "max" for the reduce operation. With "sum", you can also provide per_sample_weights giving you a weighted sum.
Why is per_sample_weights not allowed for the "max" operation? Looking at how it's implemented, I can only assume there is an issue with performing a "ReduceMean" or "ReduceMax" operation after a "Mul" operation. Could that be something to do with calculating gradients??

p.s: It's easy enough to turn a weighted sum into a weighted average by dividing by the sum of the weights, but for "max" you can't get a weighted equivalent like that.

Output:
The argument per_sample_weights was only implemented for mode='sum', not due to technical limitations, but because the developers found no use cases for a "weighted max":

I haven't been able to find use cases for "weighted mean" (which can be emulated via weighted sum) and "weighted max".

Feature request: Weighted average for EmbeddingBag


Instruction: Which format is preferable for tflite model NCHW or NHWC?

Input:
For gpus nchw mode is preferrable due to speed benefits, but what is the preferable mode in terms of mobile perfomance for tflite model?
Now converting from pytorch to tflite yields a working nchw model but is this mode optimal?

Output:
The TensorFlow Lite's matrix multiplication library, for convolution 2d and so on, prefers NHWC inputs .
The TFLite converter tries to automatically transform the given NCHW weights to the corresponding NHWC weights if the given weights are constant to perform well on mobile. If the given weights are not constants, the converter will add a transpose operator after the NCHW weights to choose the NHWC's based convolution 2d algorithm for mobile.


Instruction: RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed

Input:
I get the error like the title....and I found some answers so I try retain_graph=True, but it doesn't work. Maybe my code got another problems (it occurs in loss_actor.backward(retain_grah....))
q = torch.zeros(len(reward))
q_target = torch.zeros(len(reward))
for j, r in enumerate(reward):
    q_target[j] = self.critic_network(torch.transpose(next_state[j], 0, 1), self.actor_network(torch.transpose(next_state[j], 0, 1)).view(1, 1))
    q_target[j] = r + (done[j] * gamma * q_target[j]).detach()
    q[j] = self.critic_network(torch.transpose(state[j], 0, 1), action[j].view(1, 1))
loss_critic = F.mse_loss(q, q_target)
self.critic_optimizer.zero_grad()
loss_critic.backward()
self.critic_optimizer.step()

b = torch.zeros(len(reward))
for j, r in enumerate(reward):
    b[j] = self.critic_network(torch.transpose(state[j], 0, 1), self.actor_network(torch.transpose(state[j], 0, 1)).view(1, 1))
loss_actor = -torch.mean(b)
self.actor_optimizer.zero_grad()
loss_actor.backward(retain_graph=True)
self.actor_optimizer.step()

Output:
Based on the provided info about part of your computational graph, I  assume, that loss_actor and loss_critic share some parts of it, I think its state (not sure)
state -> q --> loss_critic <-- backward 1
|
-------> b --> loss_actor <--- backward 2

to reproduce your example:
# Some computations that produce state
state = torch.ones((2, 2), requires_grad=True) ** 2

# Compute the first loss
q = torch.zeros((1))
q[0] = state[0, 0]
l1 = torch.sum(2 * q)
l1.backward()

# Compute the second loss
b = torch.zeros((1))
b[0] = state[1, 1]
l2 = torch.mean(2 * b)
l2.backward()

RuntimeError                              Traceback (most recent call last)
<ipython-input-28-2ab509bedf7a> in <module>
     10 b[0] = state[1, 1]
     11 l2 = torch.mean(2 * b)
---> 12 l2.backward()
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

Trying
...
l2.backward(retain_graph=True)

doesn't help, because you have to

Specify retain_graph=True when calling backward the first time.

here, on the first backward call (for l1)
l1.backward(retain_graph=True)


Instruction: Pytorch -> [Onnx -> tensorflow] -> tflite generates lots of redundant conv2d operators

Input:
I am converting efficientnet from onnx to tensorflow for further conversion to tflite. The conversion from onnx to tensorflow yields strange results
Onnx has 1 conv2d operator

And after converting to tensorflow and later tflite model I get this stack of convs(All of them couldn't fit in the picture, there are 32 of the in total, as the number of kernels in onnx)

I suspect the issue is with onnx-tf conversion. It seems like the conv2d output are split into 32 separate outputs, then they are processed individually on separate conv2d operator and then concatenated together.
There are 2 connected issues on their github.
First has a fix for depthwise convs but it can't be applied in my case
https://github.com/onnx/onnx-tensorflow/issues/473
https://github.com/onnx/onnx-tensorflow/issues/754
Looking at the source of https://github.com/onnx/onnx-tensorflow/blob/master/onnx_tf/handlers/backend/conv_mixin.py
They do indeed split the output produce separate convolutions and later concatenate them.
Can this split to multiple convolutions be avoided?

Output:
After some additional digging I've found the following

My convs were depthwise(conv2d is depthwise in pytorch and onnx if it has groups parameter > 1)
This bunch of convs is an inefficient way of doing a depthwise conv. To do it efficiently we need to use tf.depthwiseconv

To fix this in onnx-tf v1.7.0 you should apply a patch to onnx-tf source code posted here https://github.com/onnx/onnx-tensorflow/issues/473#issuecomment-533522917
In current master branch there is an attempt to fix the issue by detecting depthwise convs but it currently contains a bug. To fix the bug in master branch you can apply a fix I posted here https://github.com/onnx/onnx-tensorflow/issues/754#issuecomment-801775203
I've used the master branch and the fix above resulting in a small graph with depthwise conv
I've also created a fork with the fix above, so you can do
pip install git+https://github.com/Vozf/onnx-tensorflow

to apply it instead of patching on your own

It seems like the issue should be fixed in the next major release(probably 1.8.0)
Also consider using https://github.com/nerox8664/onnx2keras as it supports depthwise convolutions and moreover supports full nhwc conversion which allows removing all the transpose ops.


Instruction: Installing ROCm 4.0.1

Input:
With the latest update, Pytorch offers support for ROCm 4.0.1.
I've tried to install the latest version of ROCm using the guide on the official AMD site, but doing that I've only managed to install version 4.0.0.
As I understand at this step
wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | sudo apt-key add -

echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/debian/ xenial main' | sudo tee /etc/apt/sources.list.d/rocm.list

I should change the repository, but I can't figure out how.
Could someone help me with the exact lines that I should run in order to install 4.0.1?

Output:
It appears that the maintainers of the apt repository at https://repo.radeon.com/rocm/apt/debian/, have not yet uploaded a package for rocm 4.0.1 there.
If you need this exact version, you can use the repository at http://repo.radeon.com/rocm/apt/4.0.1/ instead as recommended in the guide you linked to. Note that this will pin your rocm version to exactly this version without any automatic updates.


Instruction: Pytorch crashes cuda on wrong line

Input:
How to see which python line causes a cuda crash down the line in Pytorch, which executes asynchronous code outside of the GIL?
Here is a case where I had Pytorch crash cuda, running this code on this dataset and every run would crash with the debugger on a different python line, making it very difficult to debug.

Output:
I found an answer in a completely unrelated thread in the forums. Couldn't find a Googleable answer, so posting here for future users' sake.

Since CUDA calls are executed asynchronously, you should run your code
with
CUDA_LAUNCH_BLOCKING=1 python script.py

This makes sure the right line of code will throw the error message.


Instruction: Pytorch Siamese NN with BERT for sentence matching

Input:
I'm trying to build a Siamese neural network using pytorch in which I feed BERT word embeddings and trying to find whether two sentences are similar or not (imagine duplicate posts matching, product matching etc). Here's the model:
class SiameseNetwork(torch.nn.Module):
    def __init__(self):
        super(SiameseNetwork, self).__init__()
        self.brothers = torch.nn.Sequential(
            torch.nn.Linear(512 * 768, 512),
            torch.nn.BatchNorm1d(512),
            torch.nn.ReLU(inplace=True),
            torch.nn.Linear(512, 256),
            torch.nn.BatchNorm1d(256),
            torch.nn.ReLU(inplace=True),
            torch.nn.Linear(256, 32),
        )

        self.final = torch.nn.Sequential(
            torch.nn.Linear(32, 16),
            torch.nn.ReLU(inplace=True),
            torch.nn.Linear(16, 2),
        )

    def forward(self, left, right):
        outputLeft = self.brothers(left)
        outputRight = self.brothers(right)
        output = self.final((outputLeft - outputRight) ** 2)
        return output

bros = SiameseNetwork()
bros = bros.to(device)

Criterion and optimizer:
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(params=bros.parameters(), lr=0.001)

The training loop:
for batch in tqdm(tLoader, desc=f"Train epoch: {epoch+1}"):
        a = batch[0].to(device)
        b = batch[1].to(device)
        y = torch.unsqueeze(batch[2].type(torch.FloatTensor), 1).to(device)

        optimizer.zero_grad()

        output = bros(a,b)
        loss = criterion(output, y)
        loss.backward()

        trainingLoss += loss.item()

        optimizer.step()

Now, this seems to be working, as it produces results that are reasonable, but the validation error stops dropping at 0.13 after just a of epochs. Can't find a lot of things on this kind of NNs using Pytorch. Are there ways to optimize it? Am I doing something wrong?

Output: Your first layer is severely overparameterized and prone to overfitting (counts a total of 201 million parameters). I assume the shape 512 * 768 reflects the number of tokens times their dimensionality; if that's the case, you need to rethink your architecture. You need some sort of weight sharing or pooling strategy to reduce the num_words * dim input to a fixed representation (that's exactly why recurrent networks replaced the fully-connected varieties for sentence encoding). Specifically in transformer-based architectures, the [CLS] token (token number 0, prefixing the input) is typically used as the "summary" token for sequence- and bisequence-level tasks.


Instruction: Sorting a list of tensors by their length in Pytorch

Input:
I have a list of tensors in the form list = [tensor([1,2]), tensor([3, 4, 5])] and would like to order it in descending order based on the length of the tensors. This means the sorted list should look like list = [tensor([3, 4, 5]), tensor([1, 2])].
Using .sort(key=length) does not work, and have also tried using .sort(key= lambda x: len(x)) unsuccessfully.

Output:
You should avoid using python built-ins (list) for your variable names.
you can sort like following:
list_tensors = [torch.tensor([1,2]), torch.tensor([3, 4, 5])]

print(sorted(list_tensors, key=lambda x: x.size()[0]))

which will output :
[tensor([1, 2]), tensor([3, 4, 5])]
Or in descending order :
list_tensors = [torch.tensor([1,2]), torch.tensor([3, 4, 5])]

print(sorted(list_tensors, key=lambda x: x.size()[0], reverse=True))

output :
[tensor([3, 4, 5]), tensor([1, 2])]


Instruction: Why is this nerual network performing poorly on MNIST?

Input:
Hi I am building a nerual network in pytorch to classify MNIST and for the life of me, I can't seem to find out why this network won't perform above 7% accuracy. Any guidance would be nice.
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from keras.datasets import mnist
from keras.utils.np_utils import to_categorical
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score, confusion_matrix

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.astype("float32")/255
X_test = X_test.astype("float32")/255

X_train = X_train.reshape(X_train.shape[0],(X_train.shape[1] * X_train.shape[2]));
X_test = X_test.reshape(X_test.shape[0],(X_test.shape[1] * X_test.shape[2]));

class Net(torch.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.lin_1 = nn.Linear(784, 128)
    self.lin_2 = nn.Linear(128, 64)
    self.lin_3 = nn.Linear(64, 10)

  def forward(self,x) :
    x = self.lin_1(x)
    x = torch.relu(x)
    x = self.lin_2(x)
    x = torch.relu(x)
    x = self.lin_3(x)
    x = torch.softmax(x, dim=0)
    return x

net = Net();
loss = torch.nn.CrossEntropyLoss();
optimizer = torch.optim.SGD(net.parameters(),lr = 0.01);

X_train = torch.from_numpy(X_train);
X_test = torch.from_numpy(X_test);
y_train = torch.from_numpy(Y_train);
y_test = torch.from_numpy(Y_test)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu");
X_train.to(device);
X_test.to(device);
y_train.to(device);
y_test.to(device);
net.to(device);
loss.to(device);

y_train = y_train.type(torch.long)
y_test = y_test.type(torch.long)

net.train()
for epoch in range(10):
  #pred = torch.max(net(X_train),1);
  pred = net(X_train.to(device));

  train_loss = loss(pred,y_train.to(device));
  optimizer.zero_grad()
  train_loss.backward()
  optimizer.step()

net.eval()

pred = torch.max(net(X_test.to(device)),1)[1];
print('The accuracy for pytorch is ' , accuracy_score(y_test.cpu().numpy(),pred.cpu().numpy()));

I feel like I have to transform the data somehow. That's why I am dividing the the training and testing data by 255, and the network excepts a float for the input and a long for the output.
Here is a numpy version I made without pytorch
from keras.datasets import mnist
from keras.utils.np_utils import to_categorical
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.astype("float32")/255
X_test = X_test.astype("float32")/255

X_train = X_train.reshape(X_train.shape[0],(X_train.shape[1] * X_train.shape[2]));
X_test = X_test.reshape(X_test.shape[0],(X_test.shape[1] * X_test.shape[2]));

Y_train = to_categorical(Y_train);
Y_test = to_categorical(Y_test)
import numpy as np

print(Y_test.shape)

class DNN():
  def __init__(self, sizes, epochs=10, lr = 0.01):
    self.sizes = sizes
    self.epochs = epochs
    self.lr = lr
    self.params = self.initialization();

  def ReLu(self, x, derivative=False):
    if derivative:
      return 1. * (x > 0)
    else:
      return x * (x > 0)

  def softmax(self, x, derivative=False):
        # Numerically stable with large exponentials
        exps = np.exp(x - x.max())
        if derivative:
            return exps / np.sum(exps, axis=0) * (1 - exps / np.sum(exps, axis=0))
        return exps / np.sum(exps, axis=0)

  def initialization(self):
        # number of nodes in each layer
        input_layer=self.sizes[0]
        hidden_1=self.sizes[1]
        hidden_2=self.sizes[2]
        output_layer=self.sizes[3]

        params = {
            "W1":np.random.randn(hidden_1, input_layer) * np.sqrt(1. / hidden_1),
            "W2":np.random.randn(hidden_2, hidden_1) * np.sqrt(1. / hidden_2),
            "W3":np.random.randn(output_layer, hidden_2) * np.sqrt(1. / output_layer)
        }

        return params
  def forward (self,X_train):

    self.params["X0"] = X_train;

    self.params["Z1"] = np.dot(self.params["W1"], self.params["X0"])
    self.params['X1'] = self.ReLu(self.params["Z1"])

    self.params['Z2'] = np.dot(self.params["W2"], self.params["X1"])
    self.params["X2"] = self.ReLu(self.params["Z2"])

    self.params["Z3"] = np.dot(self.params["W3"], self.params["X2"])
    self.params["X3"] = self.softmax(self.params["Z3"])

    return self.params["X3"]

  def backpropagation (self, Y_train, output):

    update = {};

    error = 2 * (output - Y_train) / output.shape[0] * self.softmax(self.params["Z3"], derivative=True)
    update["W3"] = np.outer(error, self.params["X2"])

    error = np.dot(self.params["W3"].T, error) * self.ReLu(self.params["Z2"], derivative=True)
    update["W2"] = np.outer(error, self.params["X1"])

    error = np.dot(self.params["W2"].T, error) * self.ReLu(self.params["Z1"], derivative=True)
    update["W1"] = np.outer(error, self.params["X0"])

    return update

  def updateParams (self,update):
    for key, value in update.items():
      #print(key)
      self.params[key] -= self.lr * value

  def test_accuracy(self, X_test, Y_train):
    predictions = []
    for i in range(len(X_test)):
      output = self.forward(X_test[i])
      pred = np.argmax(output)
      predictions.append(pred == np.argmax(Y_train[i]))

    return np.mean(predictions)

  def train(self, X_train, Y_train):
        for epoch in range(self.epochs):
            print("epoch ", epoch)
            for i in range(len(X_train)):
                output = self.forward(X_train[i])
                update = self.backpropagation(Y_train[i], output)
                self.updateParams(update)

dnn = DNN(sizes=[784, 200, 50, 10],epochs=10)
dnn.train(X_train, Y_train)

print("The accuracy of the numpy network on the test dataset is ", dnn.test_accuracy(X_test,Y_test))

Output:
Well, I can immediately tell there are a couple of problems with the code you provided:

Please check the documentation for PyTorch's cross entropy loss function. If you read it, you'll notice that torch.nn.CrossEntropyLoss performs the softmax function internally. This means that you shouldn't really be using another torch.softmax as the output activation if you're using nn.CrossEntropyLoss. If for some reason you want to use softmax at the output layer, you should consider using nn.NLLLoss instead. If you look at the image I posted below, simply removing x = torch.softmax(x, dim=0) causes the loss to fall, whereas using it causes the loss to be the same (hence, bad).

You are training with way too few epochs. I tried running your code with 3,000 epochs rather than 10, and the end performance is 0.9028 rather than the original 0.1038. You can also see that the loss value drops much more compared to the original implementation (second picture).

Edit
After taking a look at your NumPy code the problem became clearer. My second point still holds in essence: you're not training your model nearly enough. I somewhat incorrectly used the term "epoch" above, but what I really meant is "steps."
If you look at your NumPy code, you have two for loops: the outer one is the number of epochs and the inner one loops through the training data. You're apparently using single-batch training for ten epochs. This implies that you're updating your model's parameters for a total of 600,000 times (60,000 training samples * 10 epochs) for the entire process. For your PyTorch code you're feeding the entire training data in one batch and training for ten epochs. This means you're updating your parameters only ten times.
If you modify your PyTorch code to be:
for epoch in range(10):
    net.train()

    for idx, _ in enumerate(X_train):
        prediction = net(X_train[idx].to(device))
        train_loss = loss(prediction.unsqueeze(0), y_train[idx].unsqueeze(0).to(device))

        optimizer.zero_grad()
        train_loss.backward()
        optimizer.step()

    net.eval()
    prediction = torch.max(net(X_test.to(device)), 1)[1]
    accuracy = accuracy_score(y_test,cpu().numpy(), prediction.cpu().numpy())
    print(f"Epoch {epoch + 1} test accuracy is {accuracy}.")

then you'll notice that it only takes two epochs for the model to reach 96% accuracy.


Instruction: Numpy/Torch : Selecting elements using indices over a dimension

Input:
Given an array like below:
np.arange(12).reshape(4,3)
Out[119]:
array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11]])

I want to select a single element from each of the rows using a list of indices [0, 2, 1, 2] to create a 4x1 array of [0, 5, 7, 11].
Is there any easy way to do this indexing. The closest I could see was the gather method in pytorch.

Output:
arr[[0,1,2,3], [0,2,1,2]]

or if you prefer np.arange(4) for the 1st indexing array.


Instruction: Fill tensor with another tensor where mask is true

Input:
I need to insert elements of tensor new into a tensor old with a certain probability, let's say that it is 0.8 for simplicity.
Substantially this is what masked_fill would do, but it only works with monodimensional tensor.
Actually I am doing
    prob = torch.rand(trgs.shape, dtype=torch.float32).to(trgs.device)
    mask = prob < 0.8

    dim1, dim2, dim3, dim4 = new.shape
    for a in range(dim1):
        for b in range(dim2):
            for c in range(dim3):
                for d in range(dim4):
                    old[a][b][c][d] = old[a][b][c][d] if mask[a][b][c][d] else new[a][b][c][d]

which is awful. I would like something like
    prob = torch.rand(trgs.shape, dtype=torch.float32).to(trgs.device)
    mask = prob < 0.8

    old = trgs.multidimensional_masked_fill(mask, new)

Output:
I am not sure what some of your objects are, but this should get you to do what you need in short order:
old is the your existing data.
mask is the mask you generated with probability p
new is the new tensor that has elements you want to insert.
# torch.where
result = old.where(mask, new)


Instruction: How to save a list of pytorch models

Input:
This is a newbie question. I have trained 8 pytorch convolutional models and put them in a list called models. I can use them for prediction so they are working. I would like to save them. I can't even work out how to save one however. I tried:

torch.save(models[0].save_dict(), "test.pth")

but this gives me:

ModuleAttributeError: 'Net' object has no attribute 'save_dict'

type(models[0]) gives me __main__.Net
How can I save this list of models?

Output:
To save model weights for one model (example the 1st) you should do
torch.save(models[0].state_dict(), "test0.pth")

And to load it back you should instantiate a new object and load weights
model0 = Net() # or however you instantiate the model in the first place

model0.load_state_dict(torch.load("test0.pth"))

You can save your other models this way.
And don't forget to call model.eval() if you are using it for inference
If you want to save all models weights using a for loop
for i, m in enumerate(models):
    torch.save(m.state_dict(), f"test{i}.pth")

And to load them all, supposing your model instantiation is something like follow :
models = [Net() for i in range(8)]  # Note that you should adapt this to whatever method you use to create your models in the first place

for i, m in enumerate(models):
    m.load_state_dict(torch.load(f"test{i}.pth"))
    # if you are using it for inference uncomment the line below
    # m.eval()

and that's it your variable models is ready for use
Last but not least you can save the whole at once (but discouraged approach)
torch.save(models, "all_models.pth")

# to load back
models = torch.load("all_models.pth")

However, This will cause error if you try to load models for example from another python script where the definition of your class Net is not available


Instruction: How to load a Pytorch model when the parameters are saved as numpy arrays?

Input:
On this GitHub repo, I've downloaded the pretrained model senet50_ft.
I load it like so:
import pickle
f = open('pretrained_models/senet50_ft_weight.pkl', 'rb')
state_dict = pickle.load(f, encoding='latin1')
f.close()

The state is loaded, the Github repos also provides the SENet model Class here.
So I managed to instanciate that model:
model = senet.senet50()

Then I Tried to load the state, but I got an error:
model.load_state_dict(state_dict)

Traceback (most recent call last):
  File "...\module.py", line 982, in _load_from_state_dict
    param.copy_(input_param)
TypeError: copy_(): argument 'other' (position 1) must be Tensor, not numpy.ndarray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...\module.py", line 1037, in load_state_dict
    load(self)
  File "...\module.py", line 1035, in load
    load(child, prefix + name + '.')
  File "...\module.py", line 1032, in load
    state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
  File "...\module.py", line 988, in _load_from_state_dict
    .format(key, param.size(), input_param.size(), ex.args))
TypeError: 'int' object is not callable

I tried to convert ndarray to Tensor by doing the following:
for key in state_dict.keys():
    state_dict[key] = torch.from_numpy(state_dict[key])

But I got an another error and I think I'm not going anywhere.
I'm new to PyTorch but I suspect that this model was serialized with an old version of PyTorch. Do you know if a solution exists?

Output: They have a load_state_dict function that does what you want.